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COOPERATIVE GRADUATE SUMMER SESSIONS IN STATISTICS 


'The University of Florida, North Carolina State College, Virginia Poly- 
technie Institute, and the Southern Regional Education Board are jointly 
sponsoring a series of cooperative summer sessions in statistics. 

The third of these summer sessions will be held at North Carolina State 
College, June 11-July 20, 1956. A session is scheduled to be held at Virginia 
Polytechnic Institute in 1957 and at the University of Florida in 1958. Each 
summer session lasts six weeks, and each course carries approximately three 
semester hours of graduate credit. 

The 1956 session will be held jointly with the Institute in Quantitative 
Research Methods in Agricultural Economies, sponsored by the Social Science 
Research Council. Several statistics courses will be oriented towards economic 
applications. 

The combined faculty for the 1956 summer session and Institute at 
North Carolina State College will include: Professor В. L. Anderson, North 
Carolina State College; Professor Gertrude M. Cox, North Carolina State 
College; Professor David B. Duncan, University of Florida; Professor Alva 
L. Finkner, North Carolina State College; Dr. E . E. Grandage, North 
Carolina State College; Professor Robert J. He der, North Carolina State 
College; Assistant Professor Cleon Harrell, North Carolina State College; 
Professor Earl O. Heady, Iowa State College; Professor Clifford С. Hildreth, 

Michigan State University; Professor Jack Levine, North Carolina State 
College; Professor Robert J. Monroe, North Carolina State College; and 
Assistant Professor Walter L. Smith, University of North Carolina, 

Courses to be offered this summer are: Statistical Methods I, Statistical 
Methods II (Design of Experiments), Statistical Theory I (Probability and 
Parent Distribution), Statistical Theory II (Sampling Distributions and 
Inference), Sample Survey Designs, Advanced Analysis IT, Advanced Calculus 
for Statisties, Stochastic Processes, Econometric Methods and Linear Pro- 
gramming. Lectures on Linear Equations (Matrix Algebra) and Production 
Functions will be given in the Institute program. 


Inquiries should be addressed to: 


Professor J. A. Rigney 

Department of Experimental Statistics 
North Carolina State College 

Raleigh, North Carolina 
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Erratum in Rozeboom, W. W. and Jones, L. V., The validity of the successive 
intervals method of psychometric scaling. Psychometrika, 1956, 21, 165-183. 


On page 183, last sentence in first paragraph should read: 


"Essentially, what our present analysis has shown is that it is 
always possible to give a distributional definition to a base 
scale regardless of whether or not a scale exists which simul- 
taneously normalizes all distributions." 
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THE ADDITIVE CONSTANT PROBLEM 
IN MULTIDIMENSIONAL SCALING* 


SawuEL J. Messick} 


EDUCATIONAL TESTING SERVICE 
AND 
БКовевт P. ABELSON 


YALE UNIVERSITY 


T The problem of choosing the correct additive constant to convert rela- 
tive interstimulus distances to absolute interstimulus distances in multidimen- 
sional scaling is investigated. An artificial numerical example is constructed, 
and various trial values of the constant are inserted to demonstrate the 
effect on the multidimensional map of making a variety of incorrect choices. 
Finally, n general solution to the problem, suggested by Dr. Ledyard R 
Tucker, is presented; each of the computational steps in this solution is 
set down for easy reference. 


The various procedures that have been thus far described for multidimen- 
sional scaling of stimulus objects [Torgerson (7), Attneave (2), Klingberg (4)] 
all have involved what might be termed the “additive constant problem.” | 
A recent variation on Torgerson’s procedure [Abelson (1), Messick (5)] 
also encounters the same problem. All of these procedures set up, at an early 
stage of the analysis, a matrix S;, of scale values representing distance 
estimates. Each element, s;, , of this matrix represents the estimated psycho- 
logical distance between stimulus j and stimulus X. Due to the nature of the 
scaling procedure used, however, a constant can be added to all the distance 
estimates without affecting the validity of the scale of psychological distances. 
This could be expressed by 


Si HC = dix (j = k), а) 


where s;, is the relative distance between j and k, and cis the additive constant 
which re-locates the zero point to produce d: , the absolute distance between 
j and k. To put it another way, the distances в; are all relative; there seems 
to be no way of directly determining the zero point on the distance scale. 
The choice of a particular value of the additive constant will in theory 
affect the subsequent multidimensional mapping of the stimuli, and thus the 
investigator is faced with the problem of making the most valid choice, or, 


P This study was suppo i Ni R ch Contract N6onr- 
f t t by Office of Naval Resear J E 
270-20 and by National ры F'onndation Grant G-642 to Princeton University. 


TNow at Menninger Foundation. 
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if you prefer, the best choice according to some pragmatie criterion. The 
criterion which has been previously suggested is that of minimizing the 
dimensionality of the stimulus space. Torgerson gave an approximate solution 
for the additive constant but did not explore the nature of the problem 
deeply. The following derivation leads to an evaluation of the effect of the 
constant on the multidimensional structure. and provides the groundwork 
for a general solution to the problem. 

The most systematic analytical procedure for arriving at a multidimen- 
sional map of the stimuli from the distance matrix D is that given by Tor- 
gerson. Each @; is first squared and these squared distances are arrayed in a 
matrix. Then a B matrix is calculated from this matrix of squared distances. 
The multidimensional map is obtained by factoring the B matrix. This 
procedure can be summarized in the following equations, which embody 
some simplifications of Torgerson’s procedure. For n stimu 


li the elements 
b;, of the B matrix can be written as 
1 1 j= ха 
2b, = = Èo di +— Уд, — hdc as TA ЗД, (2) 
Е NG Tec we k=1.2 É 
YU dfe, domu 


Let the matrix Ё be the basis for the final mapping. An element f;,, of 
F represents the projection of stimulus j on axis m. The origin of this iyi 
is at the centroid of the stimuli. 


В = FF’, 


However, the d;, are not available, 
be obtained by any of the suggested sca 


(3) 
but the set of scale values s;, can 
ling methods. s;, is related tod 


š Я : inf 
j # k by (1). The distance d;; between stimulus j and itself is assumed i 
be zero; all s;; would ordinarily equal zero as well. 
dj; = s; = 0. (4) 
The constant c is to be added, then, only when j 7 k. One equation can 
express the relationships of (1) and (4). 
di = з. + c(1 — 8), (5) 
where 1 
8, = 0 for all E (6) 
and 
d 0 when јн 
i= | (7) 
„(1 when j=k, 


Now the effect of the additive constant on the subse i 
of th 1 quent analysis сап 
be assessed by substituting (5) into (2) and simplifying with the aid of (7). 
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Dr. Bert F. Green, Jr., has indicated in a letter to the authors that he has 
independently arrived at an equivalent derivation. This will show the in- 
fluence of с on the В matrix, although it will not directly show the influence 
on the final F. A numerical example to be presented will attempt to assess 
this influence. The terms of (2) are written 


di, = зї, + Usa +L — 8). (8) 
Note that (1 — 63)? = (1 — 63). Note further that (1 — 6:)six = Six » 


since when j = k, 5, = 0 and (1 — 0)sj, = sj ; when j = k, à; = land 
0—25;—-0-25;. 


1 ЕД 2 4 2с (n— De 

a de =F Dit т 208 a c. (9) 
Lyp -Lya p к= 10 
a de = 5 + Уз + e Р (10) 


i> pad eased ыу у 0 
T k m r a Tf a 


- n 
Equation (2) can now be written as 
1 2c (n — 1) 2 
р = NECS sop МЕ 704) ca 
ira. Dt OSD 


1 2 2c (n—1) 2 
ur fies ete m с 


1 2 2с t 2) 
~a DLs  Lse Wi c 


— 82, — 208 — &'(1 — 8D. 
Grouping terms according to the power of c yields 


1 
ъ= 1 Уа 1 У 6 – 2 У) 08 
k nm т 7 k 


The first four terms give the result if с is set equal to zero and are equi- 
valent to (2) with s;, = d;,. Examination of (13) shows that the effect of 
_ the additive constant is felt differentially in the diagonal and off-diagonal 

elements of B. Substitution of typical numbers for the 5; will show that the 
bj, are often insensitive to various choices of c. The value of (13), however, 
is that it will serve as a basis for a general solution of the additive constant 
problem. 
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A Numerical Example 


A numerical example will help in clarifying the role of the additive 
constant and in assessing its influence on the final scales. Eight points were 
set up in two dimensions in the shape of a square, with one point placed in 
the middle. This configuration of nine points is shown in Figure 1. The 


TABLE 1 
Dy 
sa 6. “il л = о = or 
А о ay 2 7. 1.41} 2.236 2 2.956 2.828 
B 1 0 jt 1.414 1 1.414} 2.236 2 2.256 
с 2 i 0 2.956 1.414 l 2.828 2.236 2 
D 1 1.13 2.256 o i. 2 L l.hh 2,256 
E 1.414 1 1.414 1 o 13 1.414 1 1.41 
F 2.226 1.414 1 2 1 о 2.256 1.414 1 
G 2 2.256 2.828 1 1.414 2,256 o X 2 
н 2.256 2 2.956 1.414 di 1.414 ї o 1 
т 2.828 2.256 2 2.256 1.41} 1 2 1. o 


Absolute interpoint distances for the two-dimensional configura- 
tion of Figure 1. 


absolute interpoint distances d;, for this configuration are presented in 


Table 1. A set of relative interpoint distances can be obtained from Table 1 
by utilizing the relationship of (5). The particular set of s;, chosen for this 
example is found by subtracting one from every element of D;, . This pro- 
cedure provides the set of relative interpoint distances Sj, Of Table 2. 

In an actual experiment the absolute interpoint distances d;, are not 
available. The s;, of Table 2 would represent the data and would be obtained 
from any of the suggested scaling methods. The Six Selected for this problem 
is the set for which the smallest scale value is placed equal to zero. The 
problem for the experimenter is to estimate the constant to be added to the 
Sis (Table 2) in order to obtain a set of d;, (Table 1) which minimizes the 
dimensionality of the stimulus space. The s;, could not be used directly as 
d;, by setting с = 0 because certain inconsistencies exist among these dis- 
tances, such as the combination s., = 


« = 0,5, = 0, and s, = 1, If € = 1 were 


FIGURE 1 


Two-dimensional Representation of the Absolute 
Interpoint Distances, Dae Given in Table 1 


TABLE 2 


A B б D E F G Н 1 
Е o 1 о даь 1.236 ш 1.256 1.828 
B o o йа o Дар 1.256 à 1.236 
© 1 o 1.256 ма 0 1.808 1.226 1 
р o 41} 1.236 o 1 o ањ 1.236 
E M} o Ab o o E o Jub 
F 1.236 Au o 1 o 1.236 Ah 0 
G 1 1.256 1.828 0 Ah 1.256 o 1 
Н 1.256 1 1.256 ES o Јав o 0 
І 1.888 1.2566 1 1.236 аһ 0 1 о 


the Relative interpoint distances obtained by scaling procedure, setting 
таты poles distance equal to zero. Additive constant necessary to con- 
these relative distances into absolute distances, ud isc-1. 
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added to every element of Table 2, the absolute interpoint distances of 
Table 1 would be obtained; these distances are exactly appropriate to the 
two-dimensional configuration of Figure 1. No value of the additive constant 
can be found which would yield а one-dimensional configuration. Thus 
c — listhe value which minimizes the dimensionality of the stimulus space: 
The problem of how to estimate this proper value of the constant, however, 
remains to be considered. 

In an attempt to assess the influence of the additive constant on the 
configuration of points, a B;, matrix was computed for each of the following 
trial values of the additive constant: c = 4, 3, 2, 1, 0, —1, —2. 'These seven 
B,, matrices were obtained from the S;, matrix of Table 2 by substituting 
the successive values of c into (13). Latent roots and latent vectors were 
obtained for each B;, (Table 3). АП nine roots were extracted for each matrix, 
and these values were plotted as a function of the additive constant (Figure 2). 
The “correct” value of c for this S; is unity, at which point there are two 
positive roots and seven zero roots. As c changes, the vectors generate а 
surface, so that for any two values of c, it is possible to identify corresponding 
vectors. For overestimated values of с (2, 3, 4) the large roots are still easily 
distinguished from the small ones, but for underestimated values (0, —1, —2) 
the roots corresponding to the “true” configuration get smaller and finally 
become negative. Since underestimated values of c offer this possibility of 
Ad imaginary space, it is considerably better to overestimate c than to under- 
estimate it. If the configuration is plotted for various values of c (the values 
for the plots are obtained from Table 3), it can be seen that as c is over- 
estimated more and more, the configuration becomes larger and slightly 

convex" (see Figure 3). As cis underestimated more and more the configura- 
tion becomes smaller and “concave,” but it changes shape at a much faster 
rate upon underestimation than it does upon overestimation. The configura- 
tions for с = 0 and c = 4 are given in Figure 3. With normalized characteristic 
vectors, there is smaller relative change when с is large compared to с being 
small. Since errors arising from the fallibility of scaling procedures would 
tend to make points which really lie on & straight line deviate from this 
straight line, it can be shown that the methods of estimating the additive 
constant described by Torgerson would always give an underestimate. 


A General Solution for the Additive Constant 


At the “true” solution for the additive constant there is a minimum 
number of large positive roots and the rest of the roots are Zero. With fallible 
data, however, it should be remembered that the small roots will probably 
not equal zero but will vary positively and negatively around zero. For 
any symmetric matrix the sum of all the latent roots is equal to the sum of 
the diagonal elements. If the sum of only the large latent roots is set equal 
to the sum of the diagonals, a value of c is obtained which sets ithe sum of 

i aau Еапі. a oou £ 
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ii 1&2 


The two roots correspond- 
ing to the configuration 


та 
10 
8 "i 
I 
DR as 
6 4 The seven remaining roots 2/7 28 j 
(one root is zero) Ae И” 
44 


FIGURE 2 


The Latent Roots as a Function of Trial Values of the Additive Constant 
(The value of each root from the proper row of Table 3 plotted 
against trial values of с. The Solution is at c = 1, at which 
point there are two roots equal to six and seven roots equal 
to zero.) 


the other roots equal to zero. Dr. Ledyard R Tucker suggested this solution 
and has made this general approach 


practical by contributing much in the 
way of matrix simplification. 


From (13) the diagonal elements of В» may be written as 


1/1 1 il 2 
b; -i 228% +2 У. — т b» у) 
1 1 1 
+41 Уз t> 5 з, = уз Xs) (14) 


1, 1 
+$e(1 - 2. 
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Configuration of Points 
Corresponding to c = O' 


© 


Configuration of Points Corresponding toc=h 


FIGURE 5 


Note that for j = k, У 8, = у, айй Зз» = уыз 
k H k Г 


The sum of the diagonal elements is 
25 b; = i > = ES ў 

7 n P» >; Sit — 9n 2 dish (16) 

2 1 rafı- 4) 

+2 >; M >; У) +“ = 


Simplifying, 
= 1 2 [d 1 zs 2 
Ум Et 2 pata Des 
Let, X, be the latent vector corresponding to the first latent root &i - 


BX, = BX, . (18) 


rà X, is normalized so that the sum of the squares of its elements is equal 
to unity, then, 


(17) 


XBX, = В.. (19) 
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'The sum of the large latent roots is 
У в = X{BX, + XiBX, + --- + X/BX,, (20) 


where p is the number of large principal components. 
Equation 13 may be rewritten in matrix notation as 


B=A+cHh+3¢H, (21) 


where the elements a;, , e;, , and h;; of matrices А, E, and Н, respectively, are 
-ifiyg 41 І 
an =3 (2 1 5-4-5 уз), (22) 
1 
n 


T 
€i = Уз ва аъ – 5 У Уз, (23) 
i i k 


k 


= 
o 
I 


= - (7 = №), and h; = (1 = 1 (24) 
Substituting (21) in (20) » 

Ув = (X1AX, + cXIEX, + 3c X(IH X) 

+ (Х;АХ» + eXLEX, + ic X{HX,) 

toss: (ЖАЗ, + oXIEX, + 40 XSHX,), 


Ув = 5 ХлХ, +e У xux +10 У) хүнх,. 


(25) 


(26) 


root, has equal coefficients, and the other (n — 1) : 
, — l) vectors a mi 
However, any vector which has ; Te indeterminate, 


XIHX, = By, = 1. 
У XX, = 2 B, =p (p 


i = number of roots taken). (28) 
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Therefore, (26) can be written 


Р. Р 


Ув = Y Ах, +e È ХХ, + 596. (29) 
7 7 2 


Tf the sum of the large latent roots of (29) is set equal to the sum of the 
diagonals of (17) to yield a quadratic in c, the solution of this quadratic 
equation will give the value of c which sets the sum of the small latent roots 
equal to zero. | 

The solution outlined in (17) to (29) involves several computational 
steps. In any of the suggested scaling procedures it is necessary to obtain a 
matrix of relative interpoint distances S;; which can be adjusted so that the 
smallest interpoint distance is set equal to zero. A trial value of c must then 
be selected, and it is much better to overestimate than to underestimate. 
In practice either the largest value or the average value in the S;, matrix 
could be used as atrial value. If some of the stimuli are relatively close together 
these will usually provide an overestimate of c. In any event, the value of 
the additive constant obtained in this solution will indicate whether ¢ was 
overestimated and is being approached from above, or whether it was under- 
estimated and is being approached from below. 


A Numerical Illustration of the General Solution for the Additive Constant 


In order to try out the solution, the “data” of Table 2 were selected, 
and a trial value of с = 4 was chosen. This trial value was then inserted into 
(13) to obtain a B matrix (Table 4). A principal components analysis of B 
yields the X, vectors (the two largest principal components forc = 4 are 
given in the first two columns of Table 3). It is not necessary, however, to 


TABLE h 


А B © р] Е F G H 1 

A 11.44940 2.14480 -1.05060 2.14480 — -.91850 -3.56305 -1.05060 -3.56305 -5.53340 
в “ели! 8.84020 2.24480  -.90150 -.2%090  -.90150 -3-5630 -3.65980 -3-563035 
C -1.05060 2.14480 11.44940 -3.56305 -.97850 2.14480 -5.53340 -3-563053 -1.05060 
D 21480 -.90150 -3.56305 8.84020 -.5%090 -365980 глмёф -.90150 -3:56303 
E -.97830  -.5h090 -.97830 -.5%090 6.07760 --94090 -.97830 -.5%090 -97830 
F 

в 

H 


-3.56305  -.90150 2.14480 -3.65980 -:50090 8.8020 -3.56305 -:90150 8.19480 


“1.05060 3.56505 25155340) arago, --9T8e5 -3:56300 11.4949 2.14480 -1.05060 
-3.56305 -3.65980 -3.56305  -.90150  -.5h090  -.90120 одо 8.84020 2.18480 
-5.555%0 -3.56305 -1.05060 -3.56303  -.910850 2.18480 -1.05060 2.14480 11.44940 


By, matrix based upon an additive constant of C = ye 
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extract all of the principal components—only the large ones. аи 
the centroid method is an approximation to principal axes, a centroi ana у : 
could be used if the number of factors extracted is not large. Iterations s 
this solution will then yield closer and closer approximations to denis 
components as well as closer approximations to c. The X; vectors must then 
be normalized. 


X, (normalized) = | .39805| X, (normalized) = | 398057 
42792 0 

39805 — 39805 

0 42792 
ё р 0 

0 — 49192 

— 39805 39805 
— 42792 0 

L.— 39805 | L.— .39805 | 


The values of 57,57, s?, and 25/97 з, obtained from Table 2 are then 
substituted in (17) to yield the sum of the diagonals of В. 


È bi; = 2.91956 + 5.07911c + 4c, (30) 


Matrices A and E are constructed using ( 


22) and (23). Substituting the 
appropriate numerical values into (29) give 


S 
258 = 4.56580 + 6.41801 +e. 
Equating (30) and (31) and solving, c — .997 and c — — 55, 

The graph of Figure 2 indicates that it is considerably easier to distinguish 
the large roots from the small ones when the sum of the latent roots is large. 
The value of c desired, then, is the one which will give the highest 2 В 
Substituting the two roots into (29), the desired c is seen to be .997. 

У в = 4.56580 + 6.41804 (.997) + (.997)? = 11.96 
У В = 4.56580 + 6.41804 (—.55) + (—.55)? = 1.34. 


The correct value of the additive constant for the 5,, of Table 2 is unity. 
The solution of .997 may be considered only an Approximation to this correct 
value, since the X; vectors are obtain: 
value of c. But this approximation is 
{о an iterative procedure, since the la not differ widely for 
different overestimated trial values of c (see Table 3), and the coefficients of 
the other equation involved in the solution (17) are independent of с. 


(31) 


probably adequate without recourse 
tent vectors do 
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If the obtained c is satisfactory, it can be used with (13) to produce 
another B matrix, which can then be factored to yield the final F. If a more 
exact solution for c were desired, the columns of F could be substituted for 
the X; vectors in (29) to obtain a new equation for У) B as a function of c. 
'This new expression for m 8 could then be set equal to the expression for 
the sum of diagonals of B* (17), and a closer approximation to c would be 
obtained. If it became apparent that the original trial value was an under- 
estimate, it might be necessary to re-calculate the X, vectors with а new 
trial c, because the underestimated value might have led to the choice of the 
wrong dimensions or to the addition of extraneous dimensions (see Figure 2). 

It is also possible, however, to obtain the F matrix without computing 
another B, by using the general matrix factoring solution (3), which is sum- 
marized in the following two equations: 


ERKE = Е, (32) 
where X is a matrix of the X, vectors, and K is a matrix for which 
X'BX = К/К. (33) 


_ The matrix (X'BX) is symmetric and should be factored, say by the 
diagonal or square root method (6), to obtain K’. K^! can then be computed 
and applied in (32). It was seen above that 


В = A + сЕ + icH, (21) 
ВХ = (АХ) + (ЕХ) + (НХ). (34) 


Thus, (BX) can be found without computing B. It should be pointed out 
that (AX), (EX), and (HX) are already available from the computations 
involved in (29). In practice, however, it might be desirable to compute B 
anyway, in order to evaluate residuals. 

Before summarizing this procedure, it is important to point out that the 
additive constant problem should be regarded in the light of the purpose for 
which a multidimensional scale is being constructed. The investigator might, 
on the one hand, wish to do a broad exploratory study [ef. (1)], or, on the other 
hand, he may be interested in a fairly sensitive analysis [cf. (8)]. In the latter 
case, a slight dislocation of the stimuli in the psychological space might be 
damaging, and it would be important to get a near-exact solution for c. In 
the former case, however, where only the broad general structure of the 
space is of interest, there is apt to be little or no loss in choosing с larger than 
the best value. (Note how small the distortion is in the configuration in Figure 
З force = 4) 

A very convenient, crude approximation to ¢ in this case is as follows: 
Make all of the relative interpoint distances positive by adding some arbitrary 
positive number to every element of $;. A gross approximation to the 
solution would be to ignore the additive constant problem from this point on, 
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i onsider these relative interpoint distances as absolute interpoint n 
са d proceed with the analysis. Since ће underestimation of с coul. 

we edes а inary space or to the addition of extraneous dimensions, this 
e Eb Шы number should be chosen large. If the smallest s;, is felt to 
[Ma distance judgment fairly close to a true psychological zero, the 
i ers of the s;, will be a good choice for c. If the stimuli are thought to be 
МЕТ ШУ disparate from each other, the largest вр, can be chosen т d 
However, just selecting a large value for the additive constant withou 


applying the general solution given above can also be dangerous and should 


be done only with extreme caution. It is apparent in Figure 2 that for large 


values of c, some of the small roots take on appreciable values and may thus 
be included in the selection of the large roots. Extreme care should be taken, 
then, in interpreting any dimensions obtained without applying the general 


solution for the additive constant, for some of them may have been added 
in this extraneous fashion. 


Procedure for Obtaining the Additive Constant in Multidimensional Scaling 
1. Using any of the suggested scaling procedures, 
distances. Adjust these distances so that the smallest scal 


Sj, . Also Sj; = 0. Then construct a matrix S? 
element is 82, . 


obtain a set of relative interpoint 
e value is equal to zero to produce 
red relative distances, where each 


2. Compute Me, 214. >, 254, Dish, а ЭС Xs, 
x * c i "ш: 


nee Dou eT 5,1 Уф, a д. 
3. Substitute the proper coefficients in 
4. Construct matrix A, see (22), 
5 
6. 


of squa; 


(17) to obtain the sum of th 
and matrix E, see (23). 

ally the largest entry in Sj, will suffice, 
ng the trial c into (13) or (21). 


omponents (X; vectors) of B. The centroid method 
teration of the solution will then give closer approxi- 


ever, the trial value 


e diagonals of B. 
- Select a trial value for c. Usu 
Obtain a B matrix by inserti 


- Extract the large principal с 
may be used as an approximation, T 


i Vector so that the sum of the squares of its elements is equal 
to unity. 
9. Compute the coefficients for the equation for the sum of the latent roots of B; 
see (29) 


In step 3. Solve the resulting quadratic for c, selecting 
the value of c which produces the largest ^ в. 


to obtain K’, Compute К-1 and insert it into 
(82). 
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12. Iterate this solution for c by substituting columns of F for the X; vectors in 


(29) above. Set this new equation for > 8 equal to (17) and solve for a closer approximation 
to the additive constant. Further experience with this procedure may demonstrate that 
steps 1-11 give such a close approximation to c that this iteration is unnecessary. 
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A RAPID NON-PARAMETRIC ESTIMATE OF MULTI-JUDGE 
RELIABILITY* 


Desmonp S. CARTWRIGHT 
UNIVERSITY OF CHICAGO 


А technique is presented for obtaining a rapid estimate of reliability 
between judges, with special reference to qualitative judgments. It is shown 
that reliability and discrimination are independent and that estimates of both 
are needed. A method of obtaining an independent estimate of multi-judge dis- 
crimination is developed. It is shown that the size of item-samples is specified 
by the latter method. Tests of significance for both reliability and discrimina- 
tion are described. 


Whenever judgments or ratings are made, experimenters are faced with 
the problem of obtaining the reliability of such judgments. Frequently the 
average intercorrelation among the judges is computed, despite the violations 
of basic assumptions involved. If the judgments are not made on a metric 
System, however, this procedure cannot be used anyway. Three alternative 
methods have been proposed by Kogan and Hunt (6), employing the ? 
statistic and analysis of variance. These authors remain unsatisfied with 
their methods because they still involve violations of basic assumptions. 
The procedures devised by Guetzkow (5) for estimating reliability in the 
coding of verbal material have been criticized by Tukey, who argues that 
Guetzkow incorrectly assumes that a “unit is either correctly classified 
with 100 per cent certainty, or it is classified at random” (8, p. 68). A further 
objection to Guetzkow’s procedure is that it assumes theoretical accuracy 
of classification, which in many cases may be unwarranted. Whether or not 
a particular item is correctly classified may be a meaningful or verifiable 
question only in terms of some further classification or judgment, perhaps 
that of an expert, but still a judgment. Moreover the methods are very 
time-consuming, especially for three or more judges, and require sample 
sizes of 100-150 items. This factor may become of considerable importance 
її cases where several iterations of the reliability estimate are required, 
as In the initial building of a category system, or in the training of judges 
on an established system. 

In the course of establishing a category system for analyzing the results 


,... This technique was developed i nection with research at the Counseling Center, 
University of Chicago. This A it то supported by a research grant (PHS M 903) 
from the National Institute of Mental Health, of the National Institutes of Health, Public 
Health Service. Acknowledgment is made to Dr. Lyle V. Jones, University of Chicago, for 
ba a relevant to the technique. Responsibility rests, however, entirely with 

e writer. 
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of a role-playing experiment, the writer had need of a rapid estimate of 
multi-judge reliability. Because of the difficulties involved in extant pro- 
cedures, and further because the units of verbal material were to be classified 
on à non-metrie array of categories, a rapid non-parametrie technique was 
devised. 


Introduction to the Method 


The general situation concerned is that where a number of persons or 
objects are each to be classified or rated by a number of judges in terms 
of some system. Such a system may be a set of diagnostic classes, role de- 
scriptions, or behavior ratings for persons. It may be any category set, as 
defined by Guetzkow, for verbal material; that is, “a number of classes ог 
‘pigeon-holes’ into which the units of qualitative data may be placed” 
(5, p. 48). Where ratings are used, the scale employed may be treated as а 
category set if, and only if, there is no assumption regarding the distribution 
of ratings. Thus a seven-point scale system may be said to have seven 
independent classes, named 1, 2, --- , and 7, provided all classes are equally 
likely to be used. 

The following assumptions un 


derlie the use of the technique to be 
described: (а) the items to be : 


t judged are independent; (b) the category 
set, or classification system, is finite; (c) the classes in the system are in- 
dependent; (d) the classes are equiprobable; (e) the judges operate 
independently. 


ts. If three agree tog 


ree, then there are 3( 
agreements. The number of а, 


be compared with the maxi 
C of m- 
parison leads to a coefficient indicati oe la т 
This coefficient will be called alpha. 
Method of Computing Alpha 

Given à category set containing 


and disagreements i$ 
: Г € cases varies with J P n. 
particular J, the possible cases are c und e Eor eM 


onstant over all values of K > J. For 
K < J, however, not all cases are possible. For example, if J = 5 and K = ^ 
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at least two judges must agree, so that the case of all disagree is A 
Since the judgments are made independently, it does not matter D ч *: 
fact judges first, or who agrees with whom. The concern is solely with : 
number of agreements. Each of the cases for a particular J and K is associa’ 5 
with a specific number of agreements. In Table 1, the possible pt. 
J = 5, K > 5, are described in words and in a convenient shorthan . Also 
shown are the numbers of agreements associated with the various cases. 
Table 1 deals with the cases for judgments on one item only. Отава 
sample of № items, alpha is computed as follows: (а) for every item к е 
number of agreements achieved is tabulated; (b) these entries are then 


TABLE 1 
The Cases and the Number of Agreements Associated with 
Cases Agreements 
Verbal description Shorthand 
All five agree. : 2 
Four agree; one disagrees. 4: 6 
Three agree; two disagree 
with them, but agree 4 
together. : 4 
Thiee agree; two disagree 
with them, and also 
between themselves. us : 
Two agree; two disagree 
with them, but agree 
together; one disagrees 
with all four others. HE 2 
) Two agree; three dis- 
agree with them, and 


t 1 
between themselves. 


АП five disagree. 
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summed, giving the total number of agreements between the judges а 
items; (c) this total is then entered as numerator over а denominator Ww ^ 
is the maximum possible number of agreements between J judges on 
classes over N items. The resultant ratio is alpha. A convenient tabular 
procedure is exemplified in Table 2, for which the data were taken from an 
actual experiment using the Thematic Apperception Test. Judges were 
asked to classify stories for the source of activity in the relationship between 
the hero and the other character. There were six possible categories: com- 
pletely from self (CS); primarily from self (PS); mutuality (M); primarily 
from other (PO); completely from other (CO); and, no evidence (NE). The 
experimenter felt that the assumptions for using alpha were fulfilled. 


Establishing the Probability Value of Alpha 


In Table 2, alpha is seen to be .494. In practice the p-value of alpha 
may be found by entering Table 3 under the appropriate J and K, with 
alpha rounded to two decimal places. In this instance for J = 5, K = 6, 
alpha is found to be significant at better than the .05 level. 

Use of Table 3 presupposes that the assum 
are fulfilled. It will be seen that independence 
one judge is not assumed. On the contrary, it is assumed that several judg- 
ments by one judge are not independent. Such àn assumption is based on 
the general principles of behavior. At the extremes, phenomena like response 
fixation warrant this assumption. In general, something like a “learned- 
judgment-disposition" must be posited. The concern here, therefore, 18 
with the mean judging performance of judges. More specifically, the alpha 


computed over а sample of items represents the mean case of agreement 
achieved by the judges as a group. 


The underlying theory of case 
Table 3 was prepared will be briefly 


The relative frequency of the various cases for any J and K may be 
expressed as 


number of different ways of obtaining @ particular case 
number of different ways of obtaining all possible cases i 


Since it is immaterial which category the first jud 
denominator of the fi oregoing expression becomes 


T ages (1) 

The several cases for any J апа К may be equivalently expressed а8 

the presence or absence of partitioning among the judges. The сазе of perfect 
agreement is equivalently expressed as the absence of partitioning. The 
case 4:1 for J = 5 may be expressed as the partitioning of J into two groups; 
say (a, b, c, d) and (e). The assumption that the K classes are equiprobable 


ptions for computing alpha 
е of several judgments by any 


-probability and the manner in which 
described. 


ge selects, the algebraic 
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TABLE 2 
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Example of Computing Alpha on Data Taken From a Thematic Apperception 


Test Experiment with 7-5, K-6, N=18 


Items Cases 
5:0 4:1 3:2, FRI 2:2:1 2:1:1:1 1:1:1:1:1 
1 ГА 
2 / 
3 / 
: / 
5 / 
В / 
И / 
8 / 
3 / 
10 / 
11 / 
12 y 
13 / 
14 / 
15 / 
16 / 
17 / 


Agreements 
рег case = 


Product 


Sum products = 89, Maximum agreements = 18 x 10 = 180 


Alpha = 89/180 = .494. 


is equivalent to the assumption that the judges are equally likely to select 
any of the K classes. Under these circumstances, the probability that J 


judges will select e ible K classes can 
(2, p. 26) ct exactly r of the possible А clas 


1 K! A'O7 


K E-N! n 


siste 
7.9.70 . а 


be shown to be 


(2) 
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4 - 16 
.001 


nN 
© 
© 


.005 


3, 4, and 5, arid K 
.01 


.025 


.05 


.10 


TABLE 3 


Values of Alpha Required for Various Levels of Probability when J 


For the case of perfect a. 
i greement r = 1; all i 
class. Hence the third term of (2) becomes unit ope fb as о 
reduce to У, and the remaining terms 
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reciprocal of the total number of different ways in which J judges may behave 
to obtain all possible cases, namely, the reciprocal of (1). 

Cases involving disagreement are those in which judges may be said to 
be partitioned into two or more groups. As an example, the case 2:1:1:1 for 
J = 5 may be expressed as the partitioning of five judges into four groups. 
The probability of this case for J = 5and K = 5 is given by substituting in 
(2) with r = 4, yielding 

1 51. AU 


The right-hand term of (2) has been tabulated by Fisher (2, р. 78), so 
that computation of leading rth differences of the Jth powers of natural 
numbers commencing at zero is unnecessary. Using Fisher’s table for the 
present case 2:1:1:1 for J = 5 and K = 5 the third term is found to be 10, 
and the probability for the case is found to be .38400. 

There are instances in which two or more different cases may each be 
represented as the partitioning of J judges into r groups. For J = 5and K = 5 
the cases 4:1 and 3:2 each partition J into two groups but differ as to size 
of the groups. Since (2) gives the probability for the total partition, there 

À is need to separate the respective probabilities for each case. This is done by 
the usual method of proportional parts. 

Following Whitworth (9, pp. 58-9), the number of ways in which 
J = (ш + y) judges may be divided into two groups so that one contains т 
and the other y judges is 


(к + yed. (4) 


If J is an even number, the situation may arise where x = y. For J = 4 


the case 2:2 would be one instance of this. There are six possible partitions 
of the form (а) (y): 


(ab)cd); (афф; (ad)o); 
(cd)(ab); ^ (bd)(a; Ф©(аф. 

Except for the assignment to т or y, the second three partitions duplicate 
the first three. Since the z or y assignment is irrelevant for the present 
Purposes, when х = y a correction factor must be introduced to divide out 
duplicate partitions. This may be achieved by taking the factorial of the 
number of numerically equivalent groups in the partition. Let that number 
be e. Then, for = 4 a partition of 2:2 has e = 2. For J = 5 a partition of 


2:11: has e = 3. The general expression required for dividing J = (t + V) 
judges into two groups is 


J V(alyle!). (8) 


Where е = 0, e! = 1, in which case (4) is unchanged. 
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Since the right-hand member of (2) gives the number of a s of сга. 
i 1 i d since (5) gives the number о 
j to т groups of all possible sizes, ап i | : 
E “шшш J judges into two groups of any particular e the eut 
ue of (5) over all possible group sizes for J = x + y andr = 2 will yie 
the identity 


D Луу) = 207/91, (6) 
i=l 

5 5 У E ai rtitioning 

J = 5 and K = 5 the cases 4:1 and 3:2 both entail the par i 

of J „а т = 2 groups. In both cases e = 0. Substituting in the left-hand side 
of (6) yields 5 + 10. Entering Fisher's table for A'0" /r! with r — 2 and 
J — 5, the result is 15. From (2) the total probability associated with the 
two tole is .09600. The probabilities associated with each case are therefore 


(4:1): 09600 x 5/15 


ll 


-03200 
and 


(3:2): 09600 X 10/15 


Ш 


06400. 


Where J judges are to be partitioned into r — 3 groups, (6) becomes 


T Jipuk = A07 /31. (7) 
i=] 
For J = 5andK = 5 the cases 3:1:1 and 2:2:1 both entail the partitioning 


of J intor = 3 groups. In both cases e = 2. The probability for the total 
partition is given by (2) as .48000. Substituting in (7) yields 10 T 15 = 25. 
Hence the probabilities associated with each case are 


(8:1:1): -48000 X 10/25 


Il 


.19200 
and 
(2:2:1): -48000 x 15/25 = -28800. 


From (2), in conjunction wi 
latter for any r, the exact probabiliti 
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between .60 and .40 (cf. Table 4). The p-value of that alpha would lie between 
the p-values associated with cases 2 and 3, namely, between .03200 and 
-06400. An estimate of the p-value for alpha may be made by linear inter- 
polation. For alpha — .57, in the present example, p — .03200 4- 3(.06400 — 
.03200)/20 = .03680. 

Table 3, giving the values of alpha required for various levels of prob- 


TABLE 4 


The Cases, Alpha Coefficients, and Probability Values for J=5, K=5 


Shorthand Agreements P-values 


ability for J = 3 to 5 and K = 4 to 16, was prepared on the basis of the 
argument in the foregoing paragraphs. For a particular value of J, the 
Probabilities associated with each possible case over the values of K from 
4 to 16 were computed directly and set out in tabular form. The column 
ue p-values in Table 4 is typical of this preliminary tabulation. In the pre- 
liminary tables, thirteen such columns appeared, one for each value of K. 
Values of alpha intermediate between adjacent cases were inserted into the 
preliminary table, with successive intervals of .01. Probabilities for the 
intermediate values of alpha were computed by linear interpolation between 
the p-values of adjacent cases. The interpolations were taken to five decimal 
Places. From the preliminary tables, the values of alpha having p-values at 
or slightly less than -10, .05, .025, etc., were abstracted to form Table 3. 


Sample Size for Items 
t The question of sample size is important in two ways. First, the nature 
9* Much work in psychology precludes the possibility of obtaining large 
samples, particularly for pilot reliability studies. Most experimenters cannot 
easily obtain 150 Rorschach protocols on which to try out a new category 
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system. Second, the size of samples is important for the RUN 
problems associated with judges' discrimination among the classes о m 

stem. It will be shown below that the minimum sample size required is 
ca to three times the number of classes in the system, that is, N (min.) 
3K. This minimum is required by the method of obtaining an estimate о 
judges’ discrimination. It also serves, however, as a useful lower limit of 
sample size for estimating alpha. 


Sample Size and Judge Discrimination 


There are four possible extreme situations in the relations between 
multi-judge agreement and multi-judge discrimination. These are: maximum 
agreement and maximum discrimination; maximum agreement and minimum 
discrimination; minimum agreement and maximum discrimination; minimum 
agreement and minimum discrimination. The first of these situations is 
exemplified, for J = 3, K = 4, and N = 12, by the judges achieving perfect 
agreement on all items with three items placed in K1, three in K2, three 
in КЗ, and three in K4. This situation gives rise to a theorem which states 
that for a reasonable estimate of discrimination the sample must number 
a multiple of K. To prove this theorem note that the case of perfect agree- 
ment can be equipossible among the K classes if, and only if, the sum of 
possible cases (that is, the number of items available for classification in 
any K) is the same for all K. In the technique described below, it will be 

‚ seen that, for maximum discrimination, chi-square is always zero. This is 
possible under all conditions only if the number of items is equal to a multiple 
of K. ‘ 

The situation of maximum 


agreement and minimum discrimination 
would be exemplified for J = 3, K = 4, and N = 12, by the judges agreeing 
d only in one class, say K1. 
ectly, it may be due to the 
e same to them—they all look like К]. 


In the attempt to develop a measure of 


methods of measurin: 


: i › as developed by Pitman (7), Friedman 
(3), and Wilcoxon (9) to the kind of data wi present problem of 
discrimination is concerned. In this techni 
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significant.chi-square to result from certain J and K values unless the number 
of items (or replications) is greater than 2K. Since, from the argument above, 
the number of items in the sample must be a multiple of K, the minimum 
number of items in a sample must therefore be equal to 3K. 


Estimating the Multi-Judge Discrimination 


The method of estimating the multi-judge discrimination is the method 
of analysis of variance by ranks. A convenient exposition of the method 
and formula for computing chi-square, together with a reproduction of the 
chi-square chart developed by Bliss (1), is presented by Wilcoxon (9). The 
method of preparing the data for this analysis in regard to judge discrimi- 
nation is as follows: 

A table is prepared having K X N cells. The classes of the K system 
are considered as treatments. The items of the sample are considered as 
replications. On each item the number of judges placing the item in а 
particular category or class is entered in the appropriate cell for that item 
row and class column. The entries are made for all items. Each row is then 
ranked, with the highest rank being assigned to the cell (class) having the 
largest number of judges, and the lowest rank being assigned to the cell 
having the least number of judges. Ties are scored in the usual way. The 
columns are then summed, the sums are squared, and the squares are summed. 
This last figure is entered in the equation for chi-square. 

An example of the procedure is given in Table 5. The data for this example 
are the same as those for Table 2. 

From Table 5 it is seen that the chi-square has a p-value of .183. This 
value is clearly non-significant and adequately represents the distribution 
of class selections by the judges. There is little or no concentration on one 
or a few classes, and hence the discrimination is good. In other instances, 
some concentration may appear and be manifested in a value of chi-square 
approaching significance. In such an instance, where for example the p-value 
was .075, it would be a matter for the experimenter to decide whether or 
not the discrimination is good enough, in terms of the kind of items presented 
and other factors. 

.  lngeneral, p-values between .10 and .06 will require careful consideration 
in relation to the alpha achieved by the judges for agreement. Values of 
chi-square which have a p-value greater than .10 may be taken аз indicating 
Er discrimination on the part of the judges. Values of chi-square which 
Tn iQ aue of .05 or less may be taken to represent poor discrimination. 
thus 4 і circumstances it may be possible to combine two or more classes, 

aising discrimination efficiency on à reduced number of classes. Such 
а course would require recomputation of alpha on the revised K system. 


Since alpha varies independently of judge discrimination, a high value 
terms of whether 


of alpha should always be checked for its meaningfulness in 
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TABLE 5 


Example of Estimating Multi-Judge Discrimination on Data Taken from a 


Thematic Apperception Test Experiment with J=5, K=6, N=18 


PS Rank 


M Rank PO Rank 


0 1+ 1 4 1 4 2 6 0 1+ 1 4 
2 8 ë 2 6 0 24 о 2 0 24 0 2+ 
3 0 2 0 2 3 5 1 4+ 1 4+ 0 2 
4 0 2+ 4 6 0 2+ 0 2+ 1 5 0 2+ 
5 0 3 0 3 0 3 0 3 0 3 5 6 
6 1 4 2 54 0 2 0 2 0 2 2 5+ 
7 1 4 2 6 i 4 1 4 0 1+ 0 1+ 
8 0 3 0 3 0 3 5 6 0 3 0 3 
9 1 5 4 6 0 2+ 0 24 0 24 0 24 
10 0 2+ 0 2+ 0 2+ 1 4 6 0 2+ 
11 1 4 1 4 l 4 2 6 0 l+ 0 1+ 
12 1 5 4 6 0 2+ 0 24 0 2+ 0 2+ 
13 4 6 1 5 0 24 0 2+ 0 2+ 0 2+ 
14 0 2 1 4+ 1 4+ 3 6 0 2 0 2 
15 0 3 0 3 0 3 0 3 0 3 5 6 
16 0 2 0 2 3 44 1 4+ 0 2 
17 1 4 1 4 6 0 1+ 0 l4 
18 24 4 6 24 1 5 0 2+ 


54.0 52.5 


Squares 


3844.0 6006.25 3721.0 


5041.0 2916.0 2756.25 


2 
X re 12 х Sum ( rank totals ) 


7.  3n(p+1) 
пр(р+1) 


12 х 24284.50 . 
18x6x7 


Degrees of freedom 


3x18x7 = 7.468 


Probability of x^. = 7.468 = ‚183. 
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mt я PEN 
or not the judges showed good discrimination also. If they аа huie: 
possible, though not necessary, that their degree of agreement was 

to their poor discrimination. 
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A STUDY OF SPEED FACTORS IN TESTS 
AND ACADEMIC GRADES* 


FREDERIC M. LORD 
EDUCATIONAL TESTING SERVICE 


, Speeded and unspeeded tests of vocabulary, spatial relations, and arith- 
metic reasoning were factorially analyzed, together with certain reference 
tests and academic grades. Lawley’s maximum likelihood method was used, 
the computations being carried out on the Whirlwind electronic computer. 
Four different speed factors were isolated, together with a second-order general 
speed factor. Consistent small positive correlations between the academic 
grades and the speed factors were found. 


The speed with which an examinee responds to the items in a test 
frequently affects his score. Almost all achievement and aptitude tests are to 
some extent measures of “speed.” Tests for factor analyses are frequently 
speeded because many tests must be given in a limited time. 

Much remains to be learned about “speed,” in spite of the fact that it is 
commonly an element in test scores. Is speed on cognitive tests a unitary 
trait? Or are there different kinds of speed for different kinds of tasks? If 
so, how highly correlated are these different kinds of speed? How highly 
correlated are speed and level on the same task? How do various criteria 
relate to speed, and how speeded should tests to predict these criteria be? 
These are the questions which the present study attacks. 


Some Previous Results 


Factor analytic studies have often isolated a *perceptual-speed factor," 
usually measured by tests requiring simple, rapid visual discriminations. 
This factor is characterized by the task of (quiekly) finding in а mass of 
distracting material a given configuration which is borne in mind during the 
ord (6). Any speed test composed of very easy items is likely to have а 
oading on this factor. A more recent publication (7) breaks down “perceptual 
Speed” into at least two factors, speed of symbol discrimination" and "form 


"The writer is i i nd especially to 
Dr. titer is indebted to Dr. John French, to Dr. David Saunders, а 
of uuedyard R Tucker for helpful suggestions ‘and theoretical advice throughout the course 
udy. The active cooperation of Dr. William Shields, Educational Advisor, an i 

author js rs at the United States Naval Academy at Annapolis has been inval m y uhe 

ийуу ТУ grateful to Dr. Р. Youtz and Dr. C. W. Adams for the о; portunity o де 
I, a high-speed computer sponsored by the Office of Naval esearo ‚кш 
Не also шн for help is programming and in puttin the program on E gor ne 
Harper of "hes to express his deep appreciation to Dr. Habert Brogden and Л s Бе, 
fon helpful а Adjutant General's One for the opportunity to use their matrix rota 

ptu! guidance in its operation. 
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perception," the former relating to familiar symbols, the latter to unfamiliar 
som factors related to speed include finger dexterity, fluency of os 
pression, ideational fluency, reaction time, speed of association, speed o 
judgment, tapping, word fluency (6). Speed of closure and motor speed p 
included in (7). Rimoldi (20) finds a "speed of judgment," а speed o 
cognition," and a second-order "personal tempo" factor; but his subjects, 
like those in many earlier studies, were to work ata“ 
speed rather than at the maximal speed required by mo 

Since many tests in factor analytic studies are 5р 
factors are speed factors, although not always so descr 
the "number" factor, which is commonl 
of addition, subtraction, multiplication, 
be referred to as the mumber-speed factor. 

In spite of the presence of b 


natural, congenial" 
st tests. 

eeded, many of the 
ibed. An example is 
y measured by highly speeded tests 
and division. This factor will here 


and have yielded conflicting evidence (3, 4, 17, 18 
For further consideration of "speed factors," 
(24, pp. 80-85) and to the 33 references in (8). 


‚ 21, 22). 
' the reader is referred to 


Data for the Present Study 
The Subjects 


АП measures in this Study were obtained on 
class at the United States Naval Academy at An 


cases was used to obtain clear] 


The Tests 


649 students in the entering 


napolis. This large number of 
y interpretable results, 


The study centers arou 
and of arithmetic reasoning, 
areas. 


nd tests of the 


verbal factor, of spatial ability, 
because of the 


widespread use of tests in these 

In each area, seven tests were admin; 
sions examination, denoted by (A), which is only slightly speeded. The 
remaining six were short experi 


three tests were 


» аз follows: LSMSLS. The 
dedness that Would be required, 
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Six reference-factor tests (number, perceptual speed, word flueney) also 
were administered. These are designated by (R). 
A more complete description of all the tests follows. 


1. Word Fluency (R). The examinee writes as many words and their 
opposites as he can in four minutes. This test was included so as to determine 
its relation to the verbal factor and to the verbal-speed factor, if such were 
found. 


2. Verbal (A). This test contained both word-analogies and “double- 
definitions" items. The latter item type is essentially a sentence with two 
missing words to be selected from alternative pairs of words provided, thus 
producing a simple definition of one of the missing words. 


3, 4. Vocabulary (L). These tests require finding among the choices a 
Word opposite in meaning to the given key word. Also 5. Vocabulary (M) and 
6, 7, 8. Vocabulary (S). [9. Vocabulary (LIA) is merely the “‘last-item- 
attempted score" on test 7, to be discussed below.] 


10. Spatial Relations (A) contained block-counting and “identical- 
blocks" items. The latter require the examinee to indicate which of five 
drawings represents a key block drawn from a different angle. 


11, 12. Intersections (L). These tests require the examinee to visualize 
the two-dimensional outline of the intersection of a solid geometric object 
cut by a plane. Also 13. Intersections (М) and 14, 15, 16. Intersections (5). 
[17. Intersections (LIA) is merely the “last-item-attempted score" on test 15.] 


18. Mathematics (A) is composed of arithmetic reasoning, algebra, and 
geometry items. 


19, 20. Arithmetic Reasoning (L) consist entirely of the usual arithmetic- 
reasoning items, Also 21. Arithmetic Reasoning (M) and 22, 23, 24. Arithmetic 
Reasoning (S). [25. Arithmetic Reasoning (LIA) is merely the "lastcitem- 
attempted score” on test 28.] 


26, 27. Number Speed (R) are highly speeded reference tests for the 
subnet speed factor. 26 consists of simple addition and division, 27 of easy 
Subtraction and multiplication. 


28, 29, 30. Perceptual Speed (R) are reference tests for the perceptual- 
Speed factor, 28, Cancellation requires the examinee to cross out as many 
letter A'sina paragraph as he can in two minutes. 29. Picture Discrimination 
requires him to indicate which of three very sketchily drawn faces is different 
from the other two. 30. Number Checking requires him to indicate whether 
two multi-digit numbers are the same or different. 
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TABLE 1 


Background Information and Data 
on Speededness for the "Experimental" Tests 


Speeg. Number Test- Items Рег cent* of 
popas of ing per Examinees 
ge um E Finishing 


uli L 15 T 129 9T 

> iE ed м 3 5 5 т1 

6,7,8 ы 5 75 5 900 2 

Intersections i 15 20 hs 98 

i - UT M 20 12 100 75 

14,15,16 = S 55 9 255 1l 

19,20 Arithmetic Reasoning L 10 20 30 9h 

21 ч Ы M 15 15 60 50 

22,95,9h in n S 30 10 180 4 
*the mean of two values in the case of the level tests, of three in the 

case of the speed tests. 


Table 1 summarizes the ba 
mental” tests and shows the pro 
item in each test. The speeded t 


ckground information about the “ехрегі- 
portion of examinees who answered the last 


The three admissions tests а, 


Te composed of multiple-choice items 
few cases, eight) a 


lternative responses, The score obtained 
for each test was the number of items answered correctly 


The eighteen experimental vocabulary, intersections, and arithmetic- 
Teasoning tests were all composed of five-choice items and were scored number- 
right minus one-fourth-number-wrong. This “correction for guessing” was 


made in order that any speed factor that might be found should not be open 
to the challenge that it was merely a * o-guess-wildly" factor. It 


uncorrected scores 


designated as variables 37, 38, and 39 

The score on each of the Six 
answers, because this is the usual 
any other way, 


method f. 
they might no longer repres 
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In addition to the regular score, a “last-item-attempted score” (LIA) 
for one speeded test in each of the three areas gives a crude measure of rate- 
of-work. Inclusion of such scores in the present study was considered desirable, 
although in general the study is primarily concerned with the type of scores 
normally used in work with aptitude tests. The statistical method used to 
deal with the experimental dependence of these scores and of the NR scores 
on the other scores obtained from the same tests will be outlined later. 


School Grades 


During their first year, all students at Annapolis normally receive 
grades in each of the following: 


31. English Composition and Literature. 

32. Foreign Language. (Each student selects one of several available.) 
33. Engineering Drawing and. Descriptive Geometry. 

34. Chemistry. 


35. Mathematics. (Plane trigonometry, college algebra, plane and solid 
analytic geometry, and calculus.) 


36. Conduct. (The method by which these grades are assigned need no 
concern us, since no factor loadings of interest were found for this variable.) 

In the present study, each numerical grade is Ё averaged qvas Nue 
Semesters. Each semester course grade represents а combination of day-to-day 
course work and final-examination performance weighted in the ratio of three 
to two. The instructors could not have had knowledge of the test scores, with 
the possible exception of the three admissions tests. 

The final examinations were virtually unspeeded, almost every student 
finishing. The day-to-day work in class varied but was not in general com- 
pulsorily speeded. It is not known whether students felt pressed for time 
While doing their homework assignments. 


Statistical Analysis 
Nor malizing 


All variables were normalized before product-moment M ry ee 
computed. This was considered desirable since otherwise any speed aid 
that might be found might conceivably have been attributed 9 E 
Common features in the shapes of the score distributions of the speeded tes 
(e.g., skewness), rather than to a real speed factor. 


The Correlations 


The use of product-moment correlations is required by € е 
tests to be described later. The correlation matrix is presented as . 
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TABLE 2 
Matrix of Intercorrelations (decimal 


point omitted) 


205 560 497 568 
184 210 172 205 19 
170 18% 08% 186 138 247 1% 
249 230 172 238 1% 270 258 248 
207 1% 119 145 128 213 211 


210 258 
-082 -051 -106 -017 -064 -034 -038 


-019 -020 
346 ТоТ 665 673 72} 770 (986) 856 (745) 
097 155 099 12% 082 181 120 118 139 
216 325 257 271 295 341 562 362 368 


326 


13 


327 292 seo 101 
143 


127 102 099 271 
790 (977) 825 (625) 


552 340 566 297 
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TABLE 2 (cont.) 
Matrix of Intercorrelations (decimal point omitted) 


—— —— 


0 ei 22 23 2h 25 | 26 27 | 28 » 26 


55 


25 


5 15% 185 174 216 255 |258 200 | 225 212 122 | 505 184 170 29 207 -082 346 097 216 
i 365 335 329 364 195 | 051 050 | 058 221 -062 | 560 210 18» 250 156 -051 | 707 155 325 
1 285 279 275 276 160 | оът оз | 052 084 -038 | 497 172 084 172 119 2106 | 665 099 257 
1 339 255 286 301 147 | 013 025 |os 085 -023 | 568 205 186 238 145 -017 | 675 ie en 
8 299 300 297 31% 19%-| 086 088 |оот 100 -o56 | 557 192 138 1% 128 -064 |72} 082 295 
о 305 357 555 380 226 |133 165 |211 245 095 | 519 186 247 270 213-03} | 770 181 3 
о 515 355 550 364 29k |196 178 | 205 25} 105 | 590 220 192 258 211 -038 (9%) 120 362 
8 308 384 347 588 290 | 205 202 |233 250 112 | io 226 221 248 210 -019 |85 118 362 
G 208 305 282 316 455 | 559 358 | 516 35} 249 | 375 20% 182 228 258 -020 (145) 159 368 


gee 2209 2}7 -016 | 120 082 476 
Т 295 289 557 257 12} |-o}6 -018 | 105 215-028 | 105 1% 56 321 328 019 |11 = 
: 297 280 280 230 112 |-068 -022 | 076 205 -059 | 095 125 582 510 288 025 |oho 698 272| 12 
à 2M 33} 338 252 105 |-o92 -о%5 | o65 231-080 | 102 100 605 318 300 056 |o87 722 317) 12 
О 209 310 345 266 192 -018 012 | 114 292 -0}8 | 075 110 57% 327 308 023 | 127 790 552| 14 
: 289 291 338 254 177 |-028 -028 | оте 285 -022 | 057 200 368 292 285 025 | 102 (977) 340 15 


ою 0-10 uv FUND HP 


32h 361 370 307 194 123 277 005 |0598 106 584 825 366 
121 17h 192 152 394 150 231 ose | 005 019 225 


260 351 302 177 308 
606 551 532 514 eii | 155 195 | 100 115 026 | 309 192 355 42k №1 ОТ? 2тт 295 507 
560 5% 548 55 209 | 207 276 | овв 129 озо | 281 228 328 429 Wi 029 |258 270 28 ® 


--- 6 518 5n eA | 187 xo | 147 155 oso | 518 тт 325 378 376 ot | 304 278 9%) D 
632 --- 632 659 393 | 320 409 |178 195 120 | 218 190 338 380 m6 085 | 351 295 629) 22 
518 652 --- 610 (X) el зол | 278 гг m8 | 272 225 350 wo ico 095 | 347 332 (909) © 
5M. 659 610 --- 349 | 329 407 | 175 i99 157 | 300 215 302 260 ioo 084 | 360 252 55| 24 
28}. 395 (454) 3à9 --- | зге 30 | зь 280 187 | 278 157 185 280 500 -о12 | 340 237 (6T) 22 
ооо 286| 26 

347 216 5 160 222 081 m "ET 


167 198 090 


152 175 207 
109 08% 551 201 216 021 
o82 127 123 165 287 07 |135 
LL. єт 300 М8 586 081 584 036 277 
175 оё 127 | 6T --- 302 k9T 50% 17 | 270 ae 


MT 118 178 175 245 
155 195 242 199 280 
090 120 108 137 187 


318 278 272 300 178 | 160 167 
177 190 225 213 157 


222 198 
515 338 350 302 183 | ові оо зоо 302 --- 604 570 177 1% 50 351) 25 
) 378 380 420 364 28 |236 рз M48 497 604 --- 806 125 259 285 427) 34 
"| 376 416 430 400 зоо воб: .--- 250 [22352125 N58) 25 


386 50} 570 
ові 147 177 125 1% --- 
58. 250 1% 
036 09% 540 
277 230 351 "T 


0 oT 085 095 Оё oho оот 082| 36 


| 278 295 332 252 257 | ooo -015 
4| 55} 629 (949) 596 (670) | 286 348 
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Variables 9, 17, 25, 37, 38, and 39, which were not used in the factoring, ees 
$ › ? ? s e E 

experimentally dependent on variables 7, 15, and ў 23. The consequen 

Dalou high correlations are placed in parentheses in the table. 


Lawley's Maximum Likelihood Method of Factor Analysis 


Factors were extracted by Lawley's maximum likelihood. method. йа" 
this important method has not often been mentioned in this country anc 
since it (or a modification) is likely to become widely used in the near пиш 
some references are listed. The basic development was given in (12). Ex- 
tensions and further developments appeared in (13, 15, 1, 2, and 23). [A 
maximum likelihood Method II, avoiding the assumption of multivariate 
normality, was developed in (13). This second method will not be considered 
here, since the usual optimum properties of 


maximum likelihood estimates 
do not appear to hold (1, 11). Whittle (25) derived a relatively simple solution 


e where the variables are of known 


papers (in English) by Bartlett, 
Lawley, and others appeared in (26). Very 

basic differences betw 

common-factor analysis 
of Lawley. 


Lawley’s meth 


een Hotelling’s pri 
and described furth 


1. The number of common factors is 
2. The procedure in effec determine 


potheses. 
5. The usual problem of estimati 

concern, since the maximum likeli 

outcomes of the procedure, 
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small correlation matrices because of the large amount of computations 
required. From a computational point of view, Lawley's method can be de- 
scribed essentially as equivalent to the task of finding the latent roots and 
vectors of a modified correlation matrix, the correlations being modified by 
dividing them by a simple function of the unknown latent vectors. 


Extraction of Factors 


The application of Lawley’s method to the actual data was carried out 
on Whirlwind I, a high-speed digital electronic computer. The computing 
program was written by the author with a view to minimizing the use of 
computer time in case convergence should require hundreds of iterations. A 
single iteration with this program required roughly 12 seconds, the time 
varying somewhat with the number (m) of factors hypothesized. 

The original hypothesis of the author suggested that m should be at least 
9 for the 33-variable matrix analyzed. However, application of Lawley's 
method to the initial set of trial values for the factor loadings failed because 
the computations generated imaginary numbers. Extremely close initial 
approximations to the solution were necessary whenever m was at all large. 

The problem was dealt with as follows. Computations were first carried 
out with m = 4. Initial trial values of the factor loadings were arbitrary 
except that (a) calculated loadings on the first centroid factor were used for 
the first column, (b) the remaining trial values were selected so that the sum 
of squares of the trial values for any one variable was equal to the highest 
correlation with that variable. The iterations were successfully completed 
form = 4. The resulting estimates of the factor loadings were used as the 
first four columns of the trial values needed to start the iterations with 
m = 5; the fifth column of these trial values was set up in accordance with 
informed guesses based on the matrix of residuals. The trial values for m = 6 
were set up in the same way from the results obtained with m = 5, and so 
forth. In every case after m = 4, the initial trial values proved to be close 
approximations to the corresponding final factor loadings. No further 
imaginary numbers were encountered. 
in of the iterative process was rapid, as 5 
Jets. ч. Table 3. The criterion used for stopping iteratio 
а, iscrepancy between the corresponding factor loa 
atleast Cessive iterations should remain less than .002 thr 

ve iterations. 


hown by the second 
n required that the 
dings produced by 
oughout ten 


he matrix of residuals obtained with each value of m was tested for 
Information about the 


signi 

n cance by means of Lawley's chi-square test. И 1 
rogress of the computations and about the chi-square significance tests is 

ae m Table 3. Although arguments could be advanced for extracting an 

eleventh factor, it was decided to stop with ten. 


'The orthogonal unrotated matrix of the maximum likelihood estimates 
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TABLE 5 


Tests of Significance and Other Information According 
to the Number ( m ) of Factors Hypothesized 


Number of Number of Sum of  Chi-Square Degrees of Probability 

UI M Le ep Latent Calculated Roue Level for 
j - те 

(n) Convergence Roots from Residuals Chi-Square 01-5948 

+ 2 61 2,605 402 < .01 

2 ES 69 895 373 < 101 

6 25 75 2 565 beat 

i 25 Д8 550 518 < 01 

s E 80 436 292 © 201 

2 22 85 557 267 el 

= 28 8 28 245 ‘07 


of the factor loadings is 
the latent root for each f. 


Estimation of Unrotated Factor Loadings for Experimentally Dependent 
Variables 


The six variables in Table 4 with loadin 
not included in the 33-variabl 
were extracted. These loadings were estimated by the method briefly outlined 
in the following paragraphs, 


The usual factor equation, R се рр’ (= is used to indicate approximate 
equality), may be written 


Q' 


SQ o. (1) 
8 Н 
where P is the 33-variable matrix of correlations 


factors, Q is the matrix of the correlation. 
with the six variables tha 


HG' = 0, 
HH' e 8, 
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TABLE 4 


Unrotated Factor Coefficients 
(decimal points omitted) 


Variable Comu- 
No. I SII Gur ту ү VI "VII VIII x X nality 
1 3k 16 09 -oh т 08 -о% -o} -07 -02 192 
2 65 Si «25 „оз т. 18 OL le Ponas 832 
5 56 51 -2h -0h -10 12 12 Ош -06  -05 672 
h & 46 .25 ло -12 08 -05 00 -12 -02 691 
2 60 55 -2% О -от 14 05 О 12 05 T 

66 %6© .16 -03 15 -о 00 о -02 05 699 
iu 68 59 їз -0p 17 -09 Ob -02 -04 оо 875 
бт 56 -12 оо 19 -19 Ol -06 оу -06 855 
9 (08) (5%) (11) (06) (58) (31) (05) (-0%) (02) (-02) (519) 
10 lg. «56 36 13 oO: 02 <2" s 05D oL 446 
i 57 -52 -22 .05 -02 11 -02 02 -l5 oT 689 
12 55 -59 -26 -03 02 10 -11 -02 -05 15 165 
i 60 -58 -e9 01 -02 -Ol -07 -03 -05 12 190 
1 © 067 б о 10 oL O5 05. ЖО ОР 
15 BB 261 231 02 1 0) "or “ie Геро oon ne 847 
16 6) 261 о (06, IL OL | 08. 05 МО а 26 
17 (50) (220) (-12) (от) (25) (-05) (28) (-02) (05) (-31) (06) 
18 be 02 2b 36 æ б <oh ДЕ ОМ ОШ 
19 59 <07 161 22 <31 <06 203," 05, 10, ыа 556 
20 25 бот ' 83 ah <29 © col Ош, От i, оо 
21 6 -02 17 37 -27 -o6 -12 00 -08 -0 62 
22. 6h .02 23 3 .15 -11 -02 -08 08 02 684 
ES 63. 209 21 35 cin мї! 022 mos REO ESSA 601 
24 61 oh 2h 59 n -05 -01 -06 o6  -0h 60% 
25 (39) (оь) (21) (29) (16) (cob) Q1) (03) (08) (- 3) (297) 
28 ah 12 5e 26 de (06... -083 Об, KONENE 
p 28 i10 55 38 A Sy бо, ч. COM RCo CaS 
28 26) „оз. 025). бу да оваа REOR A 
29 56 -09 08 o ka оо с 26 (LOL Ср 
20 Е а от 551 
31 58. 59 ^ ia өв .-09 1k -2 «19 -08 217 q8 
25 38 oh sk .-52 03 ii), Boe 5 ОВИ 2 
A Bie) Sua, йо. 26. тот», Lint ieee OO RON MOS E 
55 eb C15 2 M8 „554 219 205) 6 о2о NDS -oh Т. 
A ЖАИГА REC ОИЕ, 
ОСЫ” Hog uc ос hits 2) ЖОКПУ -09 127 
21 (66) (00) (827) 
8 (57) (-11) EO 20) (-07) (02) (-02) (-01) М 
2. (57) (-58) (-29) in e) Сш) (4) (-о2) (02) (-15) (603) 
22 (бу Có) (25) (32) Cor Co» (о) (00 toe) С 
Roots 40.92 21.90 11.28 5.10 4.09 1-34 1.08 0-85 0.65 0.58 


Roots Адо 21.90 11.28 уло mog 1Jk 1,08 0:83 046 999. 
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Since (1) and (2) never hold exactly in practice, (2) үе ie 
inconsistent set of simultaneous linear equations, there being more equa p 
than unknowns. In practice, (3) is totally ignored. A least-squares (but n 
maximum likelihood) approximate Solution for (2) can be obtained (9) by 
postmultiplying both sides by G(G'G)^', the result being 


H = Q'G(G'Gy^. (4) 


It seemed more appropriate, however, and also computationally е 
in the present case where maximum likelihood procedures had been employe | 
to post-multiply (2) by S*G(G' SG)". S' being the 33 x 33 diagona 


matrix whose elements are the uniquenesses of the 33 variables in P. The 
result is 


= Q'S*G(G'S-Gy^. (5) 


A rigorous justification for (5) is not immediately available. Sufficient 
justification is apparent, however, when 


method of analysis G^g-2 is the diago 
latent roots, and further that the basic 


equation of Lawley's method can be 
Written 
G = (р – ISPR SG). (6) 
The best justification lies in the clarity of the results obtained, as will be 
Seen presently, 
Rotation 
The rotati 


on of the original factor matrix toward psychologically 
Meaningful oblique factors Was carried out on the matrix rotator at The 
Adjutant General’s Office. Extensive final rotations were made by desk 
calculator, Variables 37, 38, and 39 (scores not “corrected for guessing") 
Were not available during the rotations, 


he main guiding Princip] 
п 


Table 5 gives the orthogonal of the thirty-nine variables on 


Sed with various meanings 
T be referred to as “factor 
atrix for rotating Table 4 
mong the primary vectors. 


coefficients.” Tab 


into Table 5. Table 7 gives the intercorrelations a 
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Rotated Factor Coefficients 


TABLE 5 


(decimal points and initial zeros omitted) 


43 


vos M N Р у s qn x 

I II IH IV V VI VII ҮШІ X 

l. Word Fluency (R) 2 B. ul 6 T g їл. -5 
2. Verbal (A) г 2 : 2 r igo 0 2% до 
б. Vocabulary (1) 6 12 - 3 3 6 2 2-8 
: i L) 6 à 5 -9 7 8-4 4 13 -6 

ё 5 Н) qr $9 of 9 -5 2 -3 3 -9 10 
T. " B 65 4 2 -5 2 27 -3 O -L 1 
8. á (s) 72 1 1 - 23 1 о 2 -3 
9. á Р 67 -3 о si 5 20 x5 ee 2 
i LIA) 38 1 -3 т 428 5 б O. 2 

10. Spatial Relations (A y 6 2 6 1 -5 -5 20 
UL. Intersections 2 (a) k 6 2 “5 k 2 5 а 6-1 
15: " L) - n $ Hub OF == 2 
dp. ° x -2 67 5 0 -7 oT o 49-0 2 
15. i 5 -1 67 о 0-2 220 ~ 2 -2 
16. 8 5) -2 69 -1 --1- 30 -2 11 
7 | 5) +6 о к= LO 5 б 2 
d LIA) 3 29 -2 - 5 -2 19 > @ 3 
18. Mathematics (А) 6 25: 01 mox б 
195 Arithmetic Reasoning (L) 5 i 22 -9 д E 8 7 8 -8 
21. " i: L -4 ok hs i à kT 6 -3 -17 
22. ü » M a 05 i Тт o- =6 T 
25. " " S тю 27-9 9 3 -8 -5 5 
2h. ii $ s) -1 о 46 о 6 2 15 о о -2 
25. " js 5) 6238 16 4 3 T baled 
5 Ü LIA) 8 1 16 8 5 2 20 à -5 5 

26. Number Speed (R) wea 6 iu 
M " (R) 355 wh 2-5 Фоо 
28. Cancellation (R) 10 -3 $ 15) = 
29. Picture Discrimination JT E E: A 2 & -5 - 10 
20. Number Checking (R) dau a = E 2 22 о i 6 6 -8 
31. English (G) >» # 5d 36 35 1 
35. Eoreign Language (G) dd зот 253. 
sh Е & E. Draw. & Des. Geom (б) 6 36 8 -8 5 4 -6 38 -h 25 
55: Meet ) 6 1 26 JB d а $9 6j; 2 -2 
56. c thematies (G) #5 2 eB A =k Е Ө 66 -7 -12 
* Conduct (G) 5 т 6 2 25: o 315448 21 

21+ Vocabula. 2 1 

о d 
28. Intersectiows | йл 92155 216 
39. Arithne NR) a 6 -2 1 ES: 
tic Reasoning (NR) 2 2 ke 2 8 0 16 2 
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TABLE 6 


Transformation Matrix 


I ла ад I v VI VII VIII IX X 


I M5 32h 192 020 009 109 055 181 011 007 
II 637 -561 -ohT 008 030 11 -119 -056 015 005 
HII -389 -Ш8 352 129 058 -060 -078 497 000 -086 
IV -308 -036 337 295 055 -031 082 -784 -0o82 002 
ү 132 362 -510 161 167 29h og, -192 005 03} 
VI 275 08 -№6 525 тә -684 030 051 172 -020 
VII 007 002 -029 056 -350 059 532 
VIII -0X9 -189 557 -497 650 -345 -112 026 


TX 261 -ol2 -336 X7} -409 -259 O81 185 -520 888 

Х 015 207 -223 346 -493 168 -811 -145 -497 -092 

ae s ют ә 
TABLE 7 


Correlations between Primary Vectors 


digit coefficients Th igi ients 
mayb : DN x à . the 1-digit, coefficie 
У be conveniently dis - Each 2-digit coefficient without 


ustom. i 7 n 
-30 or .20, say, as not reliably different f S ее m pee e 
factor coefficients have not b 
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Interpretation of Factors 


The first three factors of Table 5 correspond to the three aptitude areas 
about which the present study is centered. They are “level” factors, in con- 
trast to the next four, which are speed factors. The eighth and to a large 
extent the ninth factors are determined by academic grades. The tenth and 
last factor seems to have no simple interpretation. All these factors will now 
be discussed in more detail. Я 

Factor I(V) is the verbal factor. In addition to the experimental vocabulary 
tests, the following variables have two-digit coefficients for this factor, as 
would be expected: 


2. Verbal Test (A) үз 

31. English Grade E 
1. Word Fluency (R) E 

32. Foreign Language Grade xr 


Factor II (S) is а space factor. In addition to the experimental inter- 
sections tests, the following variables have two-digit coefficients for this factor, 
88 would be expected: 


10. Spatial Relations (A) 1 / n 
33. Engineering Drawing and Descriptive Geometry Grades E. 
29. Picture Discrimination (R) i 


The picture discrimination test is a. reference test for the pee iaip 

factor, but the test obviously requires also the ability to perceive an 

criminate spatial patterns. n. h 
Factor III (M) is a mathematical-reasoning factor. In RO UE 

experimental arithmetic-reasoning tests, the following variables һа 

digit coefficients for this factor, as would be expected: 


18. Mathematics (A) E: 
d 35. Mathematics Grade 26 
34. Chemistry Grade у 


; се 

Factor IV (N) is the mumber-speed factor, determined by the p du 
5 included for this purpose. The only other variables wi ANNE 
Coefficients for this factor are two of the speeded arithmetic-reasonné 


test, 


17, 16 
22, 24. Arithmetic Reasoning (speeded) 


à three 
Factor у (P) is the perceptual-speed. factor, поюс C fou 
reference tests included for this purpose. No other variables h& 


coefficients for this factor. 


О A lysis 
Factor VI @) is clearly the verbal-speed factor that the present analysı 
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was designed to isolate (if it actually existed) and to study. All the two-digit 
coefficients for this factor are listed below: 


8, 7, 6. Vocabulary (speeded) .39, .30, .27 
9. Vocabulary (last item attempted) .28 
37. Vocabulary (speeded; number-right score) .28 
28. Cancellation (R) .10 


The cancellation test is a reference test for the perceptual-speed factor. The 
coefficient of .10 for this test оп the verbal-speed factor is not large enough 
to be of interest; a positive coefficient might be expected, however, in view of 
the fact that this test requires rapid work with alphabetical and verbal 
symbols. 

Factor VII (s) is clearly the spatial-speed factor that the present study 


was designed to isolate (if it actually existed) and to study. All the two-digit 
coefficients for this factor are listed below: 


17. Intersections (last item attempted) 49 
38. Intersections (speeded; number-right score) .32 
15, 16, 14. Intersections (speeded) :30, .23, .20 
25. Arithmetic Reasoning (last item attempted) .20 
39. Arithmetic Reasoning (speeded; number-right score) 16 
23. Arithmetic Reasoning (speeded) A8 


The fact that all of the speed scores on the arith 
small positive loadings on the spatial 
that the arithmetic-reasoning tests 


metic-reasoning tests have 
-speed factor is consistent; with the fact 


| contain a considerable proportion of 
simple geometry and other items that involve graphic illustrations, these 


being printed in the test booklets alongside the items. 


Factor VIII (G) is an academic-grades factor. No variables other than the 

six academic grades have two-digit coefficients for this factor. 

| Factor IX (H) appears to be some Sort of verbal-academic-grade factor, 
as indicated by its two-digit coefficients, which 


are as follows: 
31. English Grade 


‚85 

32. Foreign Language Grade 24 
4. Vocabulary (level) 13 
1. Word Fluency (В) 11 


Factor X (X) does not suggest any ready interpretation, 


The Correlations among Factors 

The correlations among the 
interest. First, it should be poin 
factor was arbitrarily set appro 
the verbal-speed factor, since it 


primary vectors in Table 7 are of paramount 
ted out that the reference axis for the verbal 
ximately orthogonal to the reference axis for 
was felt that interpretation would be hindered 
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by a choice of reference axes that would give the speeded verbal tests loadings 


of zero on the verbal factor. For the same reason, the spatial-factor axis was 


set roughly orthogonal to the spatial-speed axis. In each of these cases, the 
and the corresponding level factor 


correlations between the speeded tests 
are therefore represented approximately by the factor coefficients of the 
Speeded tests and not by the corresponding near-Zero correlation in Table de 
Because of the considerable indeterminacy аз to the proper position of 

the primary vector for the ninth factor, the verbal-factor axis and the 
academic-grades axis were both set approximately orthogonal to the axis for 
the ninth factor. 

| The mathematical-reasoning factor shows correlations of 44 and .49 
With the verbal and spatial factors, respectively. These correlations are 
reasonable in view of the fact that the arithmetic-reasoning tests include 
Verbally presented problems, geometry P and other graphically 
presented problems. The only other correlations in Table 7 as large as these 
are between various speed factors. In fact, the main thing about Table 7 is 
the consistently positive intercorrelations of the four speed factors that 
have been isolated. In general, these correlate much more highly with each 
other than they do with the three “Jevel” factors, thus demonstrating ше 
existence of a second-order general speed factor. 


The Relation of Grades to Speed hi 
The academic-grade is seen from Table 7 to be positively cor- 

related with all n of артй. factors. The ninth factor, however, whic! 
is determined mainly by grades in English and in Foreign Language, has 
Degative correlations with each of the four speed tests an 
84 factor itself. In order to interpret the 16 
s es speed factors, it is necessary 
ch of the grades with the primary vector 


.  Asshown in Table 8, each of the course grades, with one I. 
18 positi a d factors. Although these 
Positively correlated with each of the four spee ae ve relation betwee 


relationships are not high, there is clear evidence of а P 


&rades at Annapolis and speed. 


TABLE 8 


mary Vectors 
Correlations between Course Grades and Pri 


Vo Sus 
TELL III 
XX. En 39 08 
glish (С & 12 54 20 
ES Foreign E ы (G) 18 15 26 Z4 -00 -05 


Eng'g. Draw. & Des. Geom. (G) 09 69 1 
Chemistry (б) 17 56 2 


Mathematics (0) o8 
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Discussion 


No speed factor for the arithmetic-reasoning tests could po pure 
although the attempt was persistently made. The factor coefficients in A 
that the speeded arithmetic-reasoning tests tend to involve the E d 
speed factor, the verbal-speed factor, and the spatial-speed factor to a sligh г 
tests, as might reasonably be expected. 
the fact that test 23 behaves slightly 


22 and 24, and test 19, from the маш 
test 20. It may be that an arithmetic-reasoning factor exists in the data bu 
isso very unimportant that it was not s 


al and the spatial tests, that the LIA 


responding speed factor than are the 
corrected-for-guessing Scores on any of the three speeded tests, 


ar to the “corrected-for- 
indistinguishable from the 

With the exception of English, the academic grades all have higher 
loadings on the academic-grade factor than they do on any of the aptitude 
factors. This si i 


Summary and Conclusions 
The present Study y 


А vas designed to investi ate the exi inter- 
relations of Various speéd factor, : ie ешге and 
grades. 


® and their relation to academic course 
Speeded and unspeeded, by 


spatial ability, and arithmetic re 
students at the U, $, Naval Ac 


t otherwise 


à Parallel, tests of vocabulary; 
asoning wer 


© administered to 649 entering 
napolis. Also included in the 


FREDERIC M. LORD 49 


Extraction of factors from the 33-variable correlation matrix was carried 
out by Lawley's maximum likelihood method, the caleulations being done 
on the Whirlwind, a high-speed electronic computer. Factoring was con- 
tinued until, after the extraction of the tenth factor, a significance test on 
the matrix of residuals showed them to be no longer statistically significant. 

Rotation to psychologically meaningful oblique axes was carried out 
with the help of the matrix rotator at The Adjutant General’s Office. The 
tenth rotated factor was found to be difficult or impossible to interpret. 
With this exception, the structure of the factor matrix was found to be so 
clear that a ready interpretation existed for every factor coefficient above .09. 

As would be expected, three of the factors obtained were verbal, spatial, 
and mathematical-reasoning factors. The reference tests included in the 
battery yielded the expected number-speed factor (ordinarily called simply 
the number factor) and perceptual-speed factor. The academic grades in the 
battery were found to define not only a general academic-grade but also a 
verbal-academic-grade factor. Finally, а verbal-speed and a spatial-speed 
factor were clearly identified and distinguished from the number-speed 
and the perceptual-speed factors. No arithmetic-reasoning speed factor was 
isolated. 

The primary vectors for all four speed factors were found to be positively 
intercorrelated, demonstrating the existence of a general speed factor at the 


second-order level. | 
All correlations between course grades and the four speed factors, with 
one small exception, were found to be positive, although not large. It is to be 
concluded that speed of various kinds plays some part in the course grades 
studied, and that speededness in the admissions examinations 1s to this 
extent justified.. It would seem that tests on which 50 to 75 per cent of the 
examinees reach the last item do not involve the speed factors needed; 


apparently, only very highly speeded tests involve these factors to any 
appreciable extent. 
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OPTIMAL TEST LENGTH FOR MAXIMUM DIFFERENTIAL 
PREDICTION* 


Paur Horst 
UNIVERSITY OF WASHINGTON 


rion a method is already available for deter- 


mining the optimal distribution of testing time for a battery of predictors, 
and reliability data are available for 


assuming that intercorrelation, validity, 

predictors of arbitrary lengths. In this article a modification and generaliza- 
tion of the method is presented for the case of differential prediction involving 
a number of criterion variables. A numerical example is given to illustrate the 


method, after which the mathematical rationale is outlined. 


For the case of a single crite 


I. The Problem 


_ Та (2) the importance of techniques for predicting success differentially 
in each of a number of different activities from a single battery of predictors 
was discussed. It was assumed that intercorrelations for a large battery of 
predictor variables were available and also correlations between these 
predictors and a large number of criterion variables. The problem was to 


select from this larger battery of predictors that subset of specified size 
which would yield the maximum index of differential prediction for the 


criterion, variables. The index of differential prediction efficiency was taken 
to be a simple function of the average of the variances for the predicted differ- 
ence scores for all possible pairs of criterion variables. The larger this average 
variance the greater the differential prediction efficiency of the battery. It 
was shown that this index is equivalent to the difference between the average 
variance of the predicted criterion measures and the average of their co- 
Variances, assuming standard measures for both predictors and criteria, and 
that the predicted criteria are the “least squares” estimates. A method for 
selecting that subset of predictors of specified size which would yield the 

maximum index of differential prediction was presented. | 
The method referred to tacitly assumes that all predictors in the battery j 
take the same amount of administration time, So that all subsets of the same 
size would also take the same amount of administration time. Usually this 
will not be the case. A more general approach to the problem might be to 
start with a given battery of predictor variables and inquire how the ad- 
tract Nonr-477(08) between the University 


*Thi i т Co A у 
of Mod ee quM UN Ev en Most of the computat p ҮЛҮ ceive cut 
by Robert Dear, Charlotte MacEwan, and Donald Mills. Much cre E ue E A a 
Elizabeth Cross. Supervision of both ‘computational and штп АЗИ de T р 
by William Clemans. To each of these able contributors I am deep y grateful. 
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ministration time for each predictor should be altered so that for a specified 
over-all testing time the index of differential prediction efficiency will be L4 
maximum. This approach would allow for increasing the length of an experi- 
mental battery as well as for decreasing it. 

As a matter of fact, for the cas 
available (1) for determining the 
battery of predictors, assuming th 
data are available for predictors o 
article to present a modification 
case of differential prediction inv: 

In this presentation testing t 
the examinee for taking the test. 


€ of a single criterion a method is already 
optimal distribution of testing time for a 
at intercorrelation, validity, and reliability 
f arbitrary lengths. Tt is the purpose of this 
апа generalization of the method for the 
olving a number of criterion variables. 

ime is taken to be the time actually allotted 
+ A more complete analysis must also take 
g instructions, practice exercises, passing 


IL Numerical Example 
The predictor variables used in this example are: 


(1) Guilford-Zimmerman Aptitude Survey, 


f Part I, Verbal Compre- 
hension 

(2) Guilford-Zimmerman Aptitude Survey, Part III, Numerical Opera- 
ions 

(3) Guilford-Zimmerman Aptitude Survey, Part VII, Mechanical 
Knowledge ' 

(4) A. C. E. P 


sychological Exam 


GEOR Pa ination, Quantitative Reasoning 


ological Examination, Linguistic Reasonin 
(6) Cooperative English Test (Form OM), Usage 8 


The matrix of inter pee ри à РИС. 
нг Е Зере intercorrelations with reliabilities in the diagonal is given 


synonymously. Е 
Тһе method of solution for the new test lengths involves a series of 
successive approximations. For large numbers of predictor and criterion 
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TABLE l 


R Matrix of Predictor Intercorreletions with Reliabilities 
Substituted for Unities in the Diagonal: 


R = r-Du 

= 1 2 3 n 5 6 È 

-Z 1 .920 5159 152 281 .163 .515 2.790 
: ie 3 259 .920 .003 1369  .298 +203 1.986 
E fe T ‚152 .003 .920 .200 .1h2  -.150 1.261 
г rog .281 .369 .200 „820 .5l9 .h26 2.645 
D s -L .763 .292 .142 .5h9 .830 .628 3.204 

nglish .515 .gh3  -.150 .h26 .628 .860 2.522 


2.645 3.20} 2.522 14.414 


> 2.790 1.986 1.267 


TABLE 2 


The r Matrix of Validity Coefficients 


1 Anthropology 370 

5 . -177 091  .29h hl 

emen :317 eo 016 299 оза 0-39 r6 

i, MEME .339  .21l o08  .2 33b +323 1.156 

5 foreign Là .526 2247 -.075 .262 .%88 .52} 1.972 

Осе ng. .295 .287  -.156 1200 .232 +126 1.28% 
Бу .10h Б .09h .170 .229 .20h 1.031 

-.001 „182 +373 .336 1.938 


Т History 

-319 .16 
S моннан .287 E -.088 к 
sychology Ше +170 .285 .h09 .ho3 1.803 


10 .096 
Zoology .336  .216  .03 .38 — .355 
2.611  3.h5l 3.734 15-524 


= 

3.3 2.239 .016 2. 
2/10 t3 рр 0o02 26l HS 5373 1.552 
ous. It is probable that the 
e of high-speed computing 
putational pro- 


very labori 
d by the us 
yield more efficient com. 


variables the solution may become 
solution would be greatly expedite! 
equipment. Further research may 
cedures. 

1. The first computational step is to calculate а matrix o; from the 
matrix r? in Table 2. The elements а! of Table 3 are the corresponding 
elements of Table 2 with their column means subtracted. Hence the columns 
of Table 3 all add to zero. 

2. The next step is to compu 
The ith element is the pro 
by one minus its reliability. 
T A in Table 4. For the first element we have Ai = 1 


3. A first approximation 18 now required for the altered test lengths. 
ths to be proportional to the original test lengths. 


We assume the new test leng 
Therefore, as a first approximation to the new test lengths we take one half 


te the elements for a diagonal matrix, A. 
al length of test i multiplied 
i in row 4, labeled 


The elements for A are given In T 
25(1.00 — .92) — 2.000. 
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TABLE 3 


The a’ Matrix: Validity Coefficients Expressed 
© in Deviation Form for Each Test 


1 2 3 2 
X .023 -.047 .089 .033 -.00l. -.016 
2  -.030 „050 .01h -048 .019 .026 
3  -.008 -.013 .006 -.020 -.011 -.050 
uU 5179 .023 -.077 -001 143 +151 
5. 052 .063 -.158  ..06 -.113 „053 
6  -.163  -.08h «092 -.093 -,116 -.159 
T „032 -.055 -.003 -.079 .028 -.037 
8 -.060 „12% -.090 „089 -.009 „028 
ӨК .093  -:05h -094 02k 064 +030 
10 -.011 -.008 .029 .057 .000 -.022 
Ck .003 -.001 -.00% .001 .001 001. 
EZ 003  -.00  ..004 +001 .001 +004 
TABLE l 
Computation of 1'0} ana l'an 
First approximation: 1n E Ту Дїр 
трт 
1 2 3 5 > 
T Ck p» 
1 E 
2S 


15:5 YD 12,5 Eb 15.0. 45 7.5 


20.0 71.0 71.0 
3 їр -060 „222. 0667 “097 — .1333 ‚озо 
4 Ya 2.000 ..T29 2.400 4.140 2,550 5.600 2780 
5 i Ау 160 .160 16. .360 


+340 +280 1.460 


h. \ 

the original test lengths. Row lin Table 4 p igi 

Row 2 of the same table is one half the ins D T ^ US qat шы; 
4. Caleulate the Teciprocals of the D 


row 3 of Table 4, » elements, These аге given in 


first value is 160 = 


culated in Step 5 are add in 
diagonal elements of Table 1, and the table i "узу кре 


able is copied int left 
quadrant of Table 5. The first diagonal element is 1.080 Wadi ree 
Dot copied in. The upper 


TABLE 5 


ENS! 
Computation of (вэлэ) ас = Ly Forward Solution 


1А 2А ЗА йе SA 18. 2B 3В B 158 B TB B 9B 10B Check = 
YA 1.080 .159 .152 .281 .763 .515 .023 -.030 -.008 .179 -.052 -.163 .032 -.060 .093 -.011 2.953 

2А 1.080 .003 .369 .292 .243 -.0%Т .050 -.013 .023 .063 -.084 -.055 .12h -.05h -.008 2.145 

ЗА 1.080 .299 .1h2 -.150 .089 .01à .006 -.077 -.158 .092 -.003 -.090 .O9h .029 1.423 

hA 1.180 .5h9 .h26 .033 .048 -.020 .001 -.061 -.091 -.079 .089 .024  .057 3.006 

5A 1.170 .628 -.00% .019 -.011 .1h3 -.113 -.116 .028 -.009 .06h 


.000 3.5h5 


1.140 -.016 .026 -.050 .151 .053 -.159 -.037 .028 .030 -.022 2.806 


.950 2.146 1.427 3.005 3.544 2.802 .078 .12T -.096 


2 „Моо -.268 -.521 -.114 .082 .251 .0%5 15,878 
.9259 1 1.000 .159 .152 .281 .163 .515 .023 -.030 -.008 


.179 -.052 -.163 .032 -.060 *.093 


-.011 2.953 2.953 
.9h61 2 1.057 -.019 .328 .180 .16T -.050 .05h -.012 -.003 .O71 -:061 -.060 .133 -.068 -.006 1.711 1.711 
.9h5? 3 1.058 .166 .038 -.220 .085 .019 .007 -.102 -.1h9 .11h -.009 -.079 .080  .030 1.037 1.038 

1.0215 4 .919 .289 .275 .029 .036 -.015 -.029 2.046 -.048 -.067 .076 .008  .057 1.545 1.5% 
1.9455 5 „51р .163 -.023 .020 .001 .029 -.069 .020 .036 -.009 .005 -.009 .676 .678 
1.4430 6 


.693 -.002 .019 -.039 Obl} .070 -.Ohl -.037 .001 .009 -.023  .69h .69% 


ISUOH Ihvd 


99 
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7. We next caleulate а matrix L, by premultiplying the matrix a, by 
the inverse of the matrix in the upper left quadrant of Table 5. The compu- 
tations for the forward solution are given in the two lower quadrants of 
Table 5 and in Table 6. The back solution is given in Table 7. The procedure 
for multiplying a matrix by the inverse of a symmetric matrix is outlined in 

3). 
P 8. The second approximation to the new test lengths is computed in the 
lower section of Table 7 as follows: | 

Row a consists of the sum of Squares of column elements of the L, matrix. 
For example, the first element in row а, namely, .0626, is the sum of squares 
of the first ten elements in column 1 of Table 7. 

Row b is copied from row 4 of Table 4, 


Row с consists of the products of corresponding elements in the two pre- 


oots of corresponding entries in the pre- 
ceding line. For example, .3537 = 4/125]. The computations to the right 


of this line and designated s are obtained by dividing the over-all new testing 
i i ў ‘8823, the sum of the elements in the row. This gives 
5 = 37.7198 


Row eis a check row. Each element in the second line above it is divided 
by the element immediately above, Thus .1251 /:8587 = ‚8587. 

Row f is obtained by multiplying each element in row d by s. For 
example, the first element is 13.3415 — 3537 X 37.7198. This line gives the 
second approximation to the new test lengths, 


Row g is obtained by dividing each element in the preceding row into 
the corresponding value in row b. Ti 


Each element in row g is multi- 
f to give row h, which should 


correspond, withi '5, to row b. For example, the 


first element is 2.001 — .150 x 13.3415, 
Row 7 is obt 
reliabilities, For 


9. As will be seen, the iterati 
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TABLE 7 


Computation of (Reap Ta. = L) Back Solution 


y L Matrix 
R ES I 2 3 5 Tea hie Gr ULIO ORE Jupp 
1 .039  -.051 -074 Ob} -.0Mh  -.003 -1 O .001 .255 
г -.075 -035 3019: 021. .030 .027 -1 O -.001 .207 
3 +008  -.00k  -.005  -.005 .020  -.056 -1 о .003 .098 
4 -136  -.002 -.075 -.058 .036 .063 -l O .000 +370 
5 .031 .085  -.110  -.026  ..166 .101 -1 о .001 .519 
6 -.159 -.040 100  -.050 +058  -.059 ! -1 о .000 467 
7 .022  -.039 -.011  -.079 .087  -.053 -1 о .002 .291 
8 -.068 .102  -.087 .082  -.018 .001 -1 о .002 .358 
9 -O74  -.067 .078 -003 .006 — .013 -1 о .002 ‚2 
10 -.005 -.021 .010 -070 | -.008  -.033 -l 0 -.001 2147 
а "i, Phu .0626  .0295  .0h77  .0268 oh 34 „0256 ск I 
4.15 2.55 5.60 17.42 17.1 


SIA, 2.00 a» - г 


b 
c d Dro -1251 „0218 114} „зо 1108 .1433 
Р + En її 
а 2 Ori -3537 „1558 à .3382  .3332 .3328 .3786 1.8823 s= M yo = 1:082 
is (Dr, LUN) ' 
e Ck: 1'D, pAQ, рд) Ф 3537 .158 3382 „3332  .3329 «3786 11  . 37.7198 
Жо у QUI 
f JB =1 (D 455 13.345 5.1995 12.7568 12.5682 12.5531 14.2807 71.0000 
2 луу 
g гар -150 „131  .188  .329  .203 .392 1.393 
' -1 
k: 1'D 2.001 .720 2.398 1.135 2.548 5.598 | И 
В e ъ,20ь f э 22 ? 299 6.663 6.663 = 1 Diag. of (аа) 


с ъъ, 
i Гр, + TAM, 1.070 1.051 1.108 1.119 1.033 1.252 
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VNMIHISKOHOAXSd 


fat st Dy, = Y NE 13.2743 


' -l 
g 1 AD, 3 +151. 
1 -l 
h Ck: 1 Dy 4D, 2.004 
3 v3 
' L -l 
1 YD, *1ADy 1.071 
I 3 


TABLE 8 


-1-1 
Computation of (+40) ас = Lo Back Solution 


Lp Matrix 
1 2 3 Y 2 6 


1 051  -.053 .072 .052 -.065 .000 
2 -.083 .036 .017 .018 .0h5 .022 
3 .003 -.005 -.003 -.009 .024 = -.0hg 
n .132  -.003 -.076 -.062 .053 .052 
5 .070 .093 5.108  -.009 -.229 .100 
6 -.176 -.0%3 .100 -.057 .081 -.056 
Т .002  -.0hi -.009 -.092 2119 -.052 
8 -.066 .105 -.085 .086 -.025 .003 
9 .076  -.069 .075 .005 .007 .010 
10 -.005  -.022 .012 .072 -.012 -.028 
а 12р .0730 0326 0460  .031h .0836 .0223 
v ТА 2.0 .2 2.0 hh 2.55 5.60 
c TDA 1659  .0235 110%»  .1290  .2132  .1250 
à i Gp, at .3820 .1532 .3323 .3603 .h618 .3536 
eck: 1 Dy) AX, as A .3020 .1532 .3322 .3602 А617 .3535 


5.3236 11.5472 12.5202 16.0473 12.2874 


+135 


.720 


1.055 


.208 
2.402 


1.128 


.331 
m 


1.151 


4159 
2.552 


.989 


456 
5.603 


1.316 


5 


т. 
= s A „ры. ШЕ 
' 2.0132 ; A 
Tu 8 Е 
H 
o 
a 
4 
ск z 
2.0432 
71.000 71.000 
1.440 
6.710 6.710= 


l Digg. of 
) 


(RAD, 
ee ea eee MERE, RECHTEN DNI ЕН E 
© 


60 PSYCHOMETRIKA 


LY 
TABLE 9 
] 142 _ 
Successive Approximations to 1D, , for ту = 2% 95 Е TE. 
РР = Velue of @ for 
Bzprox'n - - i 2 З = Successive Velues 
(-5)1 Dy: 12.50 4.50 15.00 11.50 7.50 20.00 71.00 of L 
13.34 5.50 12.76 12.57 12.55 14.28 71.00 .227 
13.27 5.32 11.55 12.52 16.05 12.29 71.00 ‚23% 


.235 


L 
1 
& 
ў .236 


1 
2 

3 

X 13.23 5.20 10.98 12.47 17.62 11.51 71.01 
5 13.31 5.15 10.76 12.46 18.13 11.19 71.00 
6 


11. To compute the Successive indices of differenti 
$: , We proceed as follows: 

(a) For the index corresponding to the first 
test length multiply each element in the L, matrix in Table 7 by the corres- 


ponding element of Table 3 and sum the products, This is the first entry, 

227, in the ф column at the right of Table 9. 
(b) To get ф„ follow the sam 

Table 8 instead of L, in Table 7. 


al prediction efficiency 


approximation to the new 


e procedure except use the L, matrix in 


Sequent ¢’s by using the elements in 
ments in Table 3. 


àpproach optima] length. 
even though severa] of the t 
For example, test 5 increases from 7 to 18 mi 
20 to 11 minutes. 
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TABLE 10 


Successive Approximations to n Dy , for Ty = To = 142 


Approx'n т 5 3 = S E Value of f for 
EGG ume s OT UN NE EE Successive Values 


(1) 1 Da: 1 25.00 9.00 30.00 23.00 15.00 40.00 142.00 of L 
2 25.38 8.85 20.0} 25.55 33.62 28.56 142.00 Ly -265 

3 25.83 7.60 16.02 25.41 42.43 24.73 142.02 Ly .282 

о 25.22 hh.5h 23.73 141.99 L3 .283 


һ 26.48 7.30 “14.7 


ТАВІЕ 11 


Successive Approximations to 1D, , for m. = 25 = 2(142) = 28h 


>. 

a неле вані 38 Value of f for 
OE en 1 2 4 6 bi Successive Value 
1 Das 50.00 18.00 60.00 46.00 30.00 80.00 284.00 оГ 

31 83.57 53.64 284.00 Ly .305 


1 

2 51.51 13.62 30.14 51. 
3 50.80 10.84 22.51 51.17 97-72 46.91 284.00 Ly 2334 
9h. 99.6h 45.34 284.01 1; 2337 


4 56.32 10.53 21-24 50. 
1 Derivation 


e is developed for altering test lengths in a battery 
le correlation with a single criterion. The develop- 
ed and the procedure will be extended to 


III. Mathematica 


In (1) a procedur 
to give maximum multip 
ment of this procedure will be review: 
the problem of differential prediction. Let 
M = the number of cases, 

n = the number of predictors, 
— an (M X n) matrix of tes 
the elements of Z of the form 


2 4 scores in a battery of altered lengths with 
W =an (M x 1) vector of criterion scores 
B 


(@ — 2)/( V Ме.) 
ith elements of the form 


WI 
(w — i)/(V/ M ow); 


— an (n X 1) vector of regression coefficients for estimating W from Z, 


f intercorrelations of tests of original lengths, 
tercorrelations of tests of altered lengths, 


= an (n X n) matrix o 
efficients for the tests of original 


r 
р c an (n X n) matrix of in 
Te = ап (n X 1) vector of validity co 
lengths, | 
P. = an (n X 1) vector of validity coefficients for the test: 
lengths, 
a =an(nXn 


s of altered 


) diagonal matrix of original test lengths, 
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D, = an (n X т) diagonal matrix of altered test lengths, 
D, = D,D% be the ratio of altered to original test lengths, : 
D. — the (n X n) diagonal matrix of reliability coefficients for the tests o 
original lengths. 
Let 
= EED: — TD. 1D;*. (1) 
Let 


є = (ZB — W). (2) 


We wish to minimize e'e with the constraining condition 1’D,1 = T, 


where T is the total testing time specified for tests with altered lengths, 
and 1 is a column vector of all unit elements. 


To obtain ¢’e minimum under this condition, let 


V = ee 4 Ри, (3) 
where \ is a Lagrangian multiplier. From (2) " 
Y = (P'Z'ZB — B'Z'W — W'ZB T W'W) + 1D, 1. (4) 
From the definitions above 
Z'Z — p, (8) 
ZW =p, (6) 
WW = 1. (7) 
Substituting (5), (6), and (7) in (4) 
V = BoB — B's, — ИВ +14 мер, (8) 
In (1) it is shown that 
Pe = Er, (9) 
and 
= PU — Dy + р,р,ру'ув-\?, (10) 
where we define D, = I — D, , а diagonal 7 liability 
ае GET 5 matrix of test unrelia 
B = 8'°8 


(11) 
Substituting (9), (10), and (11) in (8) 

VEU BIS Decl D.D,D;*)g — Br, — т 

The unknowns on the right-hand side of 

Differentiating (12) with res 

pression to zero to get an extrem 


BHIAN’ Di. (12) 
(12) are 8, D, and 2. 


pect to 8' and equating the resulting eX- 
um, 


дуў 
gi. Hac DaF D,D,D;')g = 0, 


ll 
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or 
B = r — D. + D,D,Di Te- a3) 


Differentiati ; 
ifferentiating V with respect to the scalars, b; , (1,2, , n) and 


e i 1 
quating the n resulting expressions to zero 


av y — вша)/ = 0 (14) 


ax ðb; 


8 b; = в бша)". (15) 
umming these n equations 
a 2p b; = 23 bua) P/N”. 
us in matrix notation we obtain 
x^ = 1(D.D)” 6/1 Ры. 
ese n expressions аз the diagonal 


(16) 


Substituti 
СО арры for M їп (15) and collecting th 
rix D, we obtain 


1'D,1 
D, = рҳр.р)" TODE’ (17) 
W 2 
n Dg isa diagonal matrix with the 8; as diagonal elements. 
n (1) it is shown that 
/21N71 
корде) 
в = (s D, + Dal LE (18) 
pe (18) in (17) we can therefore solve for D» ; the new test lengths. The 
multiple correlation is given by 
р? = Bre + (19) 


e case of differential prediction. 


ure to th 
given above. Let 


Next we extend the proced 
to the definitions 


C . 
onsider the following additions 


N= 

We the number of criteria, 1 
— an (M x N) matrix of criterion scores whose elements are deviate 

H Scores of the form (ш; — 2)/(У M Tsi) : 

an (M x N?) matrix consisting of difference vectors for all possible 

B 


and j, including ? = J үр. 
”› regression vectors for estimating 


Il 


pairs of criterion vectors 7 
an (n X N°) matrix of “least squares 


Ш 


Н from 7, 
with the tests of original 


ee ө 
* = the (n X N) matrix of validity coefficients 


à lengths, 
е = the (n X N) matrix of validity © 


lengths. 


oefficients with the tests of altered 
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From the differential prediction procedure (2) we have 
$ = I'DA — (I'C1/N), (20) 
the index of differential prediction efficiency, where 


C = ттт, (21) 
and Р. is а diagonal matrix of the diagonals of C. 
Let 
E-ZB—H (22) 
and 
Е, =e,’ — І (23) 
where e; is a column vector of all zero elements except the ith, which is unity: 
Let 
G =(®,,Р,... LE. (24) 
Thus we have 
Н = үс (25) 
and 
Е = ZB — Wæ. (20) 
From (23), (24), and (26) postmultiplied by G and divided by 2N we obtain 
«= (EG/2N) = (ZBG/2N) — wir — (11'/N)] (27) 


since G'G = 2N[] — (L1'/N)]. It can be shown that the trace of e'e iS 
equivalent to the trace of E'E. Let 


B - 2 
" G/2N = J. (28) 
WU — (11'/N)] = И 
К. U — G1'/N)] = ¢. (29) 
t= AI E (30) 
We wish to minimize the trace of e'e with the constraining condition 
PDT = 
Let : i 
n Yateley Мы, (81) 
Ye = ell — ('/N)]. (32) 


Substituting (5), (6), (9), (10), (28), (29), (30), and (32) in (31) we obtain 


а 5-20, _ — 
V ce [J^9 (0 р, + DDD; ey _ J^y, = y + tt] 4- MD (à 


3) 
Let 
PU qe (34) 
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Let 

"т 25 , D = 
Let а, =тД — UIN) = or (35) 
TI 36 
Tet В =т= Р, (36) 
A =Й: (37) 

Substituting (34), (35), (36), and (37) in (33) we obtain 
Y = te ШВ + Ару — L'as — eb Ul + AD. 69) 


Differentiating (38) with respect to row vectors of L/ and equating the results 


to 0 we obtain 


8 =a qup ADR UT 


ог 
a, = (R + АЁ) (89) 
Differentiating (37) with respect to D, and equating the results to 0 we 


Obtain 
x = М = Di, ADs" edo ii 
b 
Where Dir’ js a diagonal matrix whose non-zero elements are the diagonal 
elements of LL’, Hence 2 
a (Du ^^. (41) 
It can be shown that EH { 
1/2 1001 = 10А) 1, 2) 
Я а 1'Рь1 
Ubstituting (42) in (41), T 9) 
| 4 
Dj (Dii: M) - TOA 
From (39) = "T 
mcs (R + ADV) % 
Let Ё. 
RA АЮ) а. ) 
Where анн 
һ (46) 
Du = FDA D. 
and 
er (47) 


* 


RES and 
successive approximations to L; 2 


Usi i 
D ng (45), (46), and (47) as a [шн torily- 


з » Continue until Ds, stabilizes satisfac 
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i zi туеп by 
Now the regression vectors for the optimal test lengths will be gi 


Y = p pes (48) 

From (9), (10), (36), (37), and (48) Й 
Y = ò (R + ADs)", . (4 

But from (35) and (39) 3 
L = (Е + Ару") — (11'/N)). (5 


From (50) 


ap- a(R he АР), = aR ЕИ AD; ')^r(11'/N). (51) 
From (49) and (51) 


Y = IL + (R + Ару (17/N)]. (52) 
Furthermore the index of differe 


ntial prediction efficiency “ф” as defined 
in (2) can be shown to be 


Ф = tr L'a.. (59) 
The procedures outlined 
mathematical development as 
:. Table 1 is given by (36). 
Step 1 is based on (35). 
Step 2 is based on (37). 
Step 3 is based on (46). 
Step 4 consists of calculating D7! from Di 
Step 5 consists of calculating AD! . 
ep 6 consists 


of caleulating the parenthetieal term on the right side of 

(45) for i = 1, 

Step 7 consists of calculating L, from (45). 

Step 8 consists of calculating D,, from (47). . 

Step 9 consists of caleulating an L, matrix from (45). 

Step 10 uses equation (47) to calculate D, . 

In general Steps 6 and 7 are Tepeated for successivo values of i in (45) 

or successive values of ¢ in (47). 

о get successive values of ¢, 


in Section II may be related to the above 
follows: 
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THE REGRESSION OF GAINS UPON INITIAL SCORES 


R. Е. GansIDE 
UNIVERSITY OF DURHAM, ENGLAND 


ins upon initial scores is 


A method of estimating the regression of gains 
hich have been used in 


suggested and compared with two other methods w! 
recent investigations. 


I. Introduction 
that the correlation between obtained 


gains and initial scores tended to be negative when the correlation between 
true gains and true initial scores was not negative. In the same year Thomson 
(10) showed that this tendency was due to the same errors of measurement 
Occurring in both gains and initial scores with different sign. He derived а 
formula for caleulating the correlation between true gains and true initial 
Scores, Thomson (11) and Zieve (16) have given alternative forms of 
Thomson's original formula. They did not, however, consider curvilinear 


regression. 

The question of the regression of gains upon initial scores has arisen 
„again recently in several papers concerned with the effect of practice and 
Coaching on intelligence test scores (7, 13, and 14). It therefore seems 
appropriate to consider the measurement of this regression, both linear 
and curvilinear. 


II. Linear Regres: 


Thorndike (12) pointed out in 1924 


sion of Gains upon Initial Scores 
or two parallel tests as defined by 


If the same test is given twice, 
Gulliksen (3, p. 11) are given to the same group of testees, then 
Y = X + G, 0 
Where X = true initial score, Y = true final score and G = true gain. Now 
(2) 


сүїхү — 9X 


ra йү iu ru з 
Tex = 
e 72 + oy — 2sxcr'xY 


and, therefore 
bax = brx — 1. (8) 
= 1. But if there js any 


It is cl 2) that if r. = 1, then rex ре 
ж н та dividuals within arrays of true EY оте 
= 1, then 


Variation in true gai ind 
gain between 1n! i) 
then neither rxy nor rox Will equal unity. If, and only if, Sexe ned I 
bex = (сү/тх) Fa 1. (9 
67 
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In general, however, box increases with rxy as well as with the ratio of с, е 
c, . Thus any estimate of the regression of gains upon initial scores shoul 
increase not only with the ratio of the standard deviations of final and initial 
scores but also with the correlation between these scores. 

In practice, what we want to do is to measure the true gains at different 
levels of initial ability as indicated by initial test scores. That is, we want 
to be able to ascertain the regression of true gains upon obtained initial 


scores, whether the regression be linear or curvilinear. We shall deal with 
linear regression first. Now 


СҮТ-ү — GxT.x 
VrYtzy ^ Ох-х 


Tg: = ; (5) 
oe 
where x = obtained initial score, and, therefore, 
СҮТ:ү — СУТ, 
bee = Per ex, (6) 
Oz 


Assuming errors of measurement to be independent of initial scores, and 
applying the usual correction for attenuation, (6) becomes 
bos = “Ёз” = бы, (7) 
0. j 
where y = obtained final score and r= reliability of initial score. Thus, 
Bas = бы та, (8) 


The value of bc. obtained from (8) inereases with T- and with the ratio of 
с, to c, . Errors of measurement do not tend to make bez negative; they do 
not appear in G. 


It should be noted that r... may not be equal to r., ; if rxy < 1, then 
UCM (9) 

where т, = reliability of final Score. Thus it is essential to obtain an in- 

dependent measure of 7: . One of the internal consistency methods must 

be used for this purpose, If 

for the variance of boz, 


The first half of the 


ula [Kelley (4, p. 406)] should be used i 
the form 


Ты, = E, (10) 


R. F. GARSIDE 69 


where C.. = covariance of z and 2, the two halves of the initial test, and 


V, = variance of the initial test. 

It should be borne in mind that the split-half method only gives accurate 
results when the two halves of the test, measure the same factor or factors, 
and when the test is a power test rather than а speed test. Cronbach and 
Warrington (2) have recently discussed the estimation of the reliability of 


time-limit tests. 
III. Variance of Regression Coefficient 


To test the significance of the linear regression of true gains upon initial 
scores, we require the variance of bo. . If the split-half method has been used 
to estimate т. , then 

View Vinc Йус„уу, — 2C atero 7 an 


The variance of 4C,../V. can be found by Pearson's method (6, p. 492). 


Ignoring cubic terms, then it follows directly from Pearson’s equation (iii) 


that 
4C, | Vaca LET 12 
Vac. =í 2) [s ү 47.0... 09 
But a3) 
Views = 167. 
and 
Oyate = AC vincet zx AC sever 62er (14) 
48Yc- 


= 4Сү,с.' Em 4C v, cat 


According to Wishart (18, P- 44), as quoted by Kelley (4, p. 555), fora normal 


bivariate population, 


wi. DOE (15) 
where V and C indicate population variance and covariance, respectively, 
idm 16) 

Waa e S067 О" = 
AAA in (15 
If N is large, we may substitute sample values for the parameters 1 ( 
ànd (16) and write (13) and (14) as follows: 
16 (V. V, + Coe) (17) 
Fage = N 
а; 
i „вту, + 800/1 09 


= (BV Ce + BV Cw 
a (у, Oo CAN 


Сус," 
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Similarly, it follows directly from another of Wishart’s equations (15) that 
Vy. = 2V2/N. (19) 
Substituting (17), (18), and (19) in (12), simplifying, and collecting terms, 
16 BN a 7 ДҮ, 0]. Q0) 
Удо, у. = MALAE + 3C;.) 4C... (V, TG. XV. + €,2] 
But 


йе у, ЧЕ 19б... (21) 
Therefore, substiuting (21) in (20), simplifying, and collecting terms, 
16V.V.. 2 а, 
У,с,,.у, = “Nye U- I V. = 2C,.. к=н | Жл" = Fotha + 2C,.r;. ] (22) 
MAAA 


тр Ver Vr — 20.11 — ты]. 


We шау note, in passing, 


that if it is assumed that V. = Vy, then (22) 
simplifies to 


de 0)" (23) 
М +.) ' 


esult as obtained by Shen (8, p. 462), 


Уде, гуу, E 

Which is the same r 
different method. 

The covariance of b,- and 4 C. 


using quite 9 


s / V, can be derived by writing 
be CV. 29 


Then, by using Pearson's equation (v) (6, p.493), 


C ecu c, irr.) = Kefu Co, Gu — Crates = Crate: + | (25) 


Now 
T 4Co, 0r + (20) 


= @ Hes + б.б). qn 
we may write 
Consens = 4(V.C,,. + С.О... + У.О 
= 4[C,..(V, + Ca) 4- C, 
Moreover, it follows from (16) that 


ve C, 0, )/N (28) 
v. + 0,)y/N. 


Сус, = 2V.C.,/N. (29) 
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"Therefore, substituting (18), (19), (28), and (29) in (25) and simplifying, 


ырс аде Videtis d 


Olors acervo = 


зау, + Cues + Coed] во) 
1 
x yi „СС. + Cue) + Ү.(Ү.б„, + V«C») 
осу, + CO + Cond] 


But 
(81) 


and collecting terms, 


Cyr б» = C., - 
multiplying out, 


) + V«C (Ve + 2С..) 


Substituting (21) and M in (30) 
EE Nar 2C. 


OTACA 7 aT 
с. + Ver бы + VV.) 


(V, — Ve) + РС= za) 


су + VrO VIV 


= у, (б + Cn) (32) 


NUT 1 y, + V-V) 


| У.С, 
= С + 
(У.С. + Vale + 27.7.)] 
ы МАЛ ЖЕЛШ + be + DI 
in (11), we get 


Thus, substituting (22) aid (32) and the usual formula for Vas, i 


167.7 90.10 — ri 
LES xim eerie ЙЛ; Ve 0-" 3 


дз BY, Ven, + bye — ben лаз 


ions that М is large and that distributions 


| Equation (33) involves the Кы + 

yd f Cramér (1, P- 366), 
| ut to me that, by а theorem 0! ; 

any Rd Lap such as au when N is large, 18 approximately 
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normally distributed about the corresponding population parameter. Thus, 
if N is large, bo. may be regarded as being distributed normally about 
bo. with a variance as given by equation (33). 


IV. Curvilinear Regression 
When the regression of true gains upon initial scores is not linear, then 
a polynomial 
б=а+ br po + + 4 pg (34) 


can be used. But before the weighting coefficients b, с, etc. can be calculated, 
we must ascertain the relationships between them and the coefficients b’, 
c’, etc. of the polynomial 


у= а + Wet ea? де аз». 4 p'r”. (35) 


These coefficients may be calculated from the available data using the 


ordinary multiple correlation technique, which Kelley (4, p. 445) refers to in 
this connection as parabolic regression. 


It follows from (6), by substituting z* for a5 


multiplying by Уш and 
correcting for attenuation, that 


of the initial test, a being an integer. 
If the two applications of the initial test are parallel, then their standard 
qual. 


deviations are e Thus, following Mollenko i 
F pf (5), as quoted by Gulliksen 
(3, pp. 119 and 120), we may write NUN И 


where z and 2 = deviation 
when 2’ is predicted from x. Now 


NC. = Уха". 


(38) 
Therefore, substituting (37) in (38), à 
NC = Tay Da"! 1 guae (89) 
Let us assume that 
Terza = 0 (40) 
From the raw score correlation formula 
Terza = (Хет? — NEIN o,o). (41) 
Thus 


7 


Ze's = Ne’ x’. 
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But 
Ne’ = Хе! = 0. 
Therefore ] d 
De'r = 0. (44) 
Thus, substituting (44) in (39), 
Goes = ta 22 /N. 
Therefore / 2 
" Cargo = т Dorn з (46) 
Һет = 1 7 А 1 
б on. less covariance between 2 and 2°. Therefore, substituting (46) 
Сог = Czy — taz Dzz° + (47) 
In matrix notation, 
bez = 02:02 . (48) 
Therefore, substituting (47) in (48), 
Bos es (у= 1,2 D4)02: › (49) 


where D,, = the row matrix [Ve € c7 С], and т, = а scalar. Thus 


bes = be te Dass - 
But * (50) 
D.Ci-n9 0- 0]. (51) 
"Therefore, reverting to ordinary notation, 
Dose T буза — un 59 
and a (52) 
(53) 


=: же 

not equal to unity. 

that, although the regression lines of 
nevertheless, the same 
s not linear, then the departure 
egression of G on x. Thus, 
{Snedecor (9, section 14.4)] 


the usual test of departure : 
of y on x provides а test of the significance of the curvilinear regression of 
G on х. If the regression of y on 2 18 significantly non-linear, it may be found 
that a second-degree polynomial wil i i 

the inclusion of z^ and higher powers of x produces any significant improve- 
ment in the prediction of y, and therefore of G, can be tested by analysis of 
variance [Kelley (4; p.448)] 


It may be argued agains 


RE 


t the method here proposed that gains should 
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с : d equivalent z values from 
the regression of z on (z + У). This was done, 
Now 
Тита) ож) = (V, — Vowa), (54) 
I Tue. = (о, + 72/2, р (55) 


E. уфа te; (56) 


T, ande = g constant, 


‹ n scatter of (y — а) against (y + 2), the 
3962 18 not changed by the substitution of E. for (y + 2), 


write anged. Therefore we may 
Now ? "a-2E, = Tones. (57) 
or, > 9:7... 4 (58) 
Thus, substituting (55) in (58), 
Og, = oor, + с.) /с. БЕД, 59) 
Now (rz) ' ( 
биза, = LLL ^ (60) 


Therefore, substituting (54), (57), and (59) in (60), 


Vs sr V 
DE QU 


bz. E 
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"Therefore 


V,/V) — 
bees = тур. (62) 


n estimate of the linear regression coefficient 
Wiseman and Wrigley. Errors of 
fficient negative and it increases 


The coefficient b, ., 7, is а 
of gains upon initial score obtained by 
measurement do not tend to make this coe 
as Т, decreases. 

Peel (7) describes the p 
the scores at given percentiles in each of 
ences between these ‘equivalent levels’ are then taken as the practice effects." 
These practice effects at each given percentile are then compared with 


corresponding initial scores. Assuming that the distributions are normal, 


the percentile method will give results as given in Table 1. 


ercentile method as follows: “After calculating 
the three distributions, the differ- 


TABLE 1 
Theoretical Differential Gains 


Final Test (y) Gain (g = y-x) 


Percentile Initial Test (x) 
98th X + 2o, ӯ +2у (ў-х) + 2(оу-оу) 
ён K+ o ӯ +9 (ӯ-х) + (су-су) 
к & У (9-9) 
16th #'- оу 379, G-3 - (сузо) 
ӯ - 20у (ў-Х) - 2(су-оу) 


We can therefore write 


je am En (63) 
с: 
and, therefore, assuming distributions are normal, 
(64) 


02:5 (с„/ с) = 1. 
will not tend to make this regression 
by the percentile method are 


to c, increases. 


It is evident that errors of measurement 
ver, gains calculated 


Coefficient negative. Moreo rae 
independent of r., but increase 28 the ratio of oy 
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If it is assumed that т = 1, then equation (62) becomes 
b, —z)E, = (es/a) — 1 
UTC (eol) +1 
= (¢,/e.) — 1 
= б. (65) 


Thus if r., is equated to unity the Wi 
methods give identical results. 


The estimates of linear regression of gains upon initial scores obtained 
by each of the three methods here 


seman-Wrigley and the percentile 


YT. Example 


Suppose the following data are obtained: c, — 12, c, 


= 10, r,, = .80 
С... = 23 (Le, r = 85), N = i 


nce of curvilinear 
-92 and, by (8), Ье. = 
eman-Wrigley method, 


using (62), the corresponding coefficient is .29 and the percentile method 


gives a coefficient of -20 [equation (64)]. 
If Tz, is .90, instead of -80, then bj, = 
coefficient = .21 and the percentile coefficient r 


methods, however, may give results dissimila; 
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ram analysis is presented that does not 


A method of Guttman scalog 
i the item response matrix. The 


involve sorting and rearranging the entries in 

method requires dichotomous items. Formulas are presented for estimating 

the reproducibility of the scale and estimating the expected value of the chance 

reproducibility, ‘An index of consistency is suggested for evaluating the repro- 
in detail. The logical basis of 


ducibility. An illustrative example is presen т 
the method is discussed. Finally, several methods are suggested for dealing 
with non-dichotomous items. 


Guttman's scaling method, known as scalogram analysis (4), has become 


popular among social scientists. However, + techniques for scalogram 
irect ith the raw data in the form 


analysis are cumbersome. They all deal direc 

of an item response matrix that has a row dent and a column for 

each item response category: An entry in the matrix indicates whether а 

particular respondent gave 2 particular item response. Various procedures 

have been described for rearranging the rows and columns of ie ae response 

matrix ini categories, 80 at a response 
, as well as for combining response g а (2) ec 


“parallelogram”? is achieved with few deviations. Suchm: ( 
e matrix 18 represented 


the scalo; ^ in whi he respons 
gram board procedure 1n which the resp 
by buckshot placed in small indentations in a set of removable CS bi 
sorting is accomplished by interchanging t their Du 1 e г и 
i tabulating the response matrix on IBM equipment Mm MEN oneil 
y Noland (1 d (2 d Kahn and Bodi PYRA 
OO) C UD 3) and Marder (8). 


methods h i Guttman (3) 9 
s have been described by Өч. ешге keen Sadgment, con- 


cerning the ki sorting likely to pay off. 

are CR Жы a imd anne to evaluate the complete Rd p 
matrix without the aid of summary statistics. For large numbers or i d 
ents, the task is overwhelming. ‘o difficult to deal wi 

10-20 items with these procedures. i 
combining items before making the scalogram analysis 
by Stouffer, Borgatta, Hays, and Henry ап. 


any Comp 
*Loi n assisted the author materially ! e mP the De tment 
is K. Anderson 3 КАА supported E part b І s us 


for this d in this ра jor 
paper. Th research reported 1n y 

oE Economics E Social Sciences at МІТ. and ш pr ont lys hnology- 

Air Force under contract with the Massachusetts ns 
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The purpose of this paper is to present a relatively simple method of 
scalogram analysis in which summary statistics are used to compute а ole 
approximation to the scale rep. (In this paper “rep” is used for re 
producibility.") In this method, which requires dichotomous items, there 18 
no limitation on the number of respondents; its application to large numbers 


of respondents giving certain Specified combinations of responses. Obtaining 
these summary statistics 18 a simple, routine, completely objective matter. 


to any statistical clerk. The method also substantially reduces the time 
required for analysis. It gains these advantages at the expense of providing 
only an approximation to the “true” rep. Certain high-order scale errors 
are ignored. However the approximation appears to be a very close one. 


The Method 

All items must be dich 

let k be the number of items, N be the number s 5, 1 

script referring to item 7 (where the items are in any arbitrary order), and 
g be a subscript referring to item g in rank order, 


sion being scaled. 
F i , the number of respondents who 
gave the positive response to the item. 
ank order according t heir popularities, 
(n:/N), with the least popular item getting rank k% phe а о popular 
item getting rank f. If there are any ties, adopt an arbitrary order. 

Step 4. Tabulate n,., = for 9=1,2,+-- k— 1. This is the number of 
respondents who gave the positive response to item g + 1 and the negative 


response to item g. If it is easier to tabulate 7751, OF n2; , , then the following 
identities can be used: ende 


fag = Nk m, H 
Dig == NG F n = Mi ~ 

Step 5. Use either of the following two methods for estimating the reP- 

A. Tabulate No+2,041,5,0-1 for g = 2,8, ... ; k — 2. This is the number 


of respondents who gave the positive response to item g + 2, and the positive 
response to item g + 1, and the negative response to item g, and the negative 
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respon: 1 å 
se to item g — 1. Estimate the rep from the formula 


oe AL k-i А 1 кз 
А Nk X Tac Nk У) т,+з,о+1.2.2—1 . 


gal -2 
B. Tab "a 
а ачаа forg = 1,2, k= 2. This is the number of 
xesmones 10 i gave the positive response to item g + 2 and the negative 
item g. Estimate the rep from the formula Е 


5-2 


рыи РЕР т > 1 
eps = 1 NÉ py Tas үр 2 батон е 


g=l 
Rep, a Г 
er - eq m should yield very similar estimates. The choice depends 
d areal s he relative ease of the alternative tabulations. Repa has the 
ge that it is known to be an overestimate of the true (sample) rep- 


The 2 ‘ 
he standard error of either Repa oF Reps is approximately given by 
Gs Rep (Rep), 


Gne» ~ 
Nk 
S . Р 
if the ги б. (Optional). Estimate the rep that would be expected by chance 
The re ems had their observed popularities but were mutually independent. 
р of independent items is estimated by the formula 


k-2 


1 k-i 1 
Rep, = 1 — xp 24 ete NE Y понтон * 
- МЕ $2 N'k o=2 
(Note that m; = N — ne) 

Compute the Index of Consistency, 


Ie Rep — Reps 
= , 
a 1 = Repr 
ere гер is either Кера ОГ Reps · The index I will be unity if the items 
cted value of zero when the items are 
the sample, 


аге per 

ы RA scalable and has an expe 

will кр If the items show some negative correlation in 
e negative. If desired, label the set of items «gealable” if T is greater 

number of items 


than .50. 
to ES 7. Give each respondent a scale score that is the 
ich he gave the positive response. 
Illustrative Example 
= 6 will be used as an 


TES E of hypothetical data with N = 2% and k 4 ] 
Tabl ple of the application of the method. The hypothetical data are shown in 
"i We 1. The tabulations for Steps 2, 5 9^ and 5B are also shown 1n Table 
pr Er qim put the items and the respon ents in rank order 1n Table 1 only to 
е an easy comparison with the usual sorting techniques. In carrying 

it is not at all necessary that the 


ou 
t the tabulations of Steps 2, 4, and 5, ! 
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TABLE 1 


Data and Tabulation for Illustrative Example 
(+= positive response; — = negative response) 


Items (i) 
2 Д 3 4 5 6 

Rank Order (g) Scale 

Respondents 6 5 4 3 2 1 Scores 


ФоУсо оломон 


+ 
+ 
+ 
+ 
+ 
+ 
+ 


TELIT ТА ПЕ i erett 
PUPP ieee deere +++ 
Yo p T3 use d Tuque) в а 
Т1++++++1 1 ++ ++++ +] + 
LAI PEI +++ #++ + +++ +++ 
© 2 к кә бә бә ©з о у> AAAA 


Step 2 

(Step )ng 
Step 6) n. 
(Step Jm. 
(Step 4) Neal, E 
(Step 5A) n 


g+2, g^, р, g-i 


(Step 5B) Dg42, z 


order. It is quite possible to Work directly w. 
sheets, or with their punched-card equivalents. 
From the formulas in Step 5, we compute 


raw data be arranged with either items or respondents in any particular 


ith the individual response 


eres 1 
Rep, = 1 It At AEBS bus L5. авт 


Lt ek 1 
Reps = 1 — 132-2 c 4-3 4-2) — 2499 (1:4 + 4-4 + 4.2) = .880- 


(The actual reproducibility is .858. The large discrepancy between this 
figure and the estimates is due to the small N in this example.) For Step б 
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we compute 


Ww 


Rep = 1 — 
2400 


(14-4 + 10-6 + 10-10 + 9-10 + 7-11) 


1 
i 555,000 0 106-4 + 9-10-10-6 + 7.9-10-10) = .826 


867 — .826 
= AINE 0 
А 1 — 826 286. 


Si У Е : A 
nce I is less than .50, the set of items 18 not “scalable.” The scale scores 


are shown in Table 1. 


Justification of ihe Method 
Ordering the items. The simplicity of the method of scalogram analysis 


n in this paper is due largely to the use of popularity to rank the items. 
uttman and his followers have used the order of items that yielded the 
i ns out to be the 


highest rep. In a large majority of the cases, 
Popularity order. In the other cases, the difference in the rep for the “best? 
pre rep) order and for the popularity. 

ing will be lost and great simplification W 
Popularity order. An arbitrary order may 
o T" is not surprising that the popu 
wi К In a perfect Guttman scale, the rank 

ith the popularity order. For imperfect 
popularity order to be “best” if the scale errors are 
inversions might be expected if items had very similar popularities but 
the effect of these inversions would be small. Very peculiar error patterns 
would be required to make the populari edly inferior to the 
best” order. 

Estimating Rep. The formulas f 
analysis of the scale errors in à pattern 
Scale, the items can be arrang 
positively to (or endorses, or ag 
to all items of lower rank order. For & fiv 5 
patterns would be possible: [TL Ils quie р --+++) 
[-==+-], [-===+} ее j.In each ideal respo 
there is a dividing point, or cut, such that all item responses to 
the cut are —, and all items responses to the right ofthe ty placing 


number of scale errors in any other respo 


a cut so that the number of +’s to the left © ЛЫ. 
to the right of the cut are minimized. АП such misplace 
4—1 would have 


errors. For example, the pattern [== : 1 
items 4 and 3, (items are numbered in decreasing order from te 


data, one would still expect the 


independent. Slight 


ep are based on ап 
. In a perfect Guttman 


its cut between 
ft to right, 16» 
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5, 4, 3, 2, 1) and one error would be counted. The pattern [~ -++ = —] could 
have its cut between items 5 and 4 or at the right of item 1. In either case 
there would be two errors. 

In order to find a rule for counting the errors in any particular response 
pattern we must consider subpatterns of responses. First, consider a pair of 
adjacent items with the response subpattern (+—); i.e., the response to 
ower ranking item is 
—. We would place the cut either to the right or to the left of this pair, and 
e would not place the cut 
between the two items, since this would yield two errors. Next, consider the 
1 à pair from a complete 
1 the complete pattern is 
pattern, for when we have 


er side of this cut. We may 
eliminating pairs of adjacent 


xample, in the response pattern 


› 2), (items are numbered 5, 4, 8, 
2, 1) leaving (—4- —); then we eliminate (4, 


are two errors in the original pattern. 


In practice the first Step is simultaneously to eliminate from the complete 


) subpatterns. These 


two items represent the second. 
when the first-order error is deleted. Th 
straddle first-order errors. In t 


-order error that appears 
at is, second-order errors always 
he same way, third-order errors straddle 
second- and first-order errors, as in (+++ =) Ot ee -L— —). In 
general, higher-order errors always straddle lower-order errors 

The formula for тер is ` 


m a 
Rep = 1 NE? (1) 


er of errors, N 1. 
Then, we have 


Rep = 1 - «E X (+-) =. +=) = 


where F is the total numb 


Н à s the number of respondents, and 
k is the number of items, 


terms of higher order. (2) 
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dose 
Now the sum of all adjacent item errors (+=) is 


k-1 
na + na + с Tea = Dy Mons ; (8) 


Similarly, 
ж, (++-—) = У То+2,0+1.2.0-1 * (4) 


2-2 
Since the higher-order error patterns occur Very infrequently, we will dis- 


regard all terms higher than second order. Substituting (3) and (4) in @) 


we obtain the formula for Repa that we presented in Step 5A. 
estimate this quantity. 


W Instead of tabulating L+), we may 
e shall assume that the two errors represented by the pattern (++— =) 


are independent. That is, the probability of а response pattern (+ 0 — 0), 
Where 0 symbolizes either response, ns 

response pattern (0 + 0 —). Then the product of 
the probability of a response pattern (++—-—). 


No+2,o+1.0.0-1 — Meri Besse. (8) 
N 1 
Hence, 
-2 1 k-2 = б 
2; ЖЕЛ r N 3 тъз. 2+1 07T * (6) 
g-2 272 
. Pr 
pubstituting (3) in (2) and substituting (6) in (4) and the result in (2) we 
ave the formula for Reps gi :n Step 5B 
pa given in Step 92- . А . "es 
ord, A cruder approximation to rep can be obiained by pe Reps. This 
seni "ns, 1 itti t term from ерл Ps + 
or patterns, 1.е., by omitting the las ee ^ DNE 


2 гох] 1 Я 2 
Pproximation may be satisfactory in some cases, 


considerabl. $ А я | 
y by including the last term. К 
d The formulas for Rep a and Reps have been checked empire ott 
published by Suchman (12, 13). the correc p 
See Rep, and Вер» for ten scales 0 
crepany is .002 for Repa and 003 for Reps - 1 
А 1 we 
а е actual reps are not necessarily those reported by A ome 
ichotomized all items and have not rejected any items 
arger numbers of items, the discrepancies may b 
Obtained here, 
The errors in the d estima 
i propose s 
ee and higher orders are neglected: It is Lage A 
E the error by assuming that scale errors ep 
ш P Mia results show that the error » a l 
The variance of Repa or Reps can : Ta 
straightforward but long, and lead to 8 complicated form 


* 
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TABLE 2 


with Correct Rep 


Empirical Comparison of Rep, and Rep, 


Scalogram No. 
to Suchm 
0 


р.126 
р.130 
р.134 
р.136 
р.138 
р.140 
p.146 
p.148 


оо у on 


ОО OY Gy 


Chance Rep. One of the major advantages of the method of Scalogram 
е rep, Rep; , can be 


tem Popularities, or 

are independent, then 
9+2,0+1,9,9=1 = To he тта М. 
Substituting these values in the formula for Rep, 7 
Rep; shown in Step 6 above, 

The hypothesis that Rep, (ог Reps) is n 
Rep; can be tested by using the variance estimate presented in Step 5. Caution 
18 necessary when such & test yields borderline Significance because of the 
many approximations involved, It should be noted that a significant rep does 
not necessarily indicate a homogeneous scale. For homogeneity the inter- 
correlations of the items should be fairly high, as well as significantly non- 
zero. It is possible to construct an index of homogeneity that will be zero 
when Rep, (or Кер,) = Rep; , and will be unity when Rep, = 1, The index 


` ыу = пот]; т, 
We obtain the formula for 


ot significantly larger than 
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^ dinge in Step 6, has these properties. It will be affected very little by 
i in the number of items, or in the item popularities. This index is very 
ч lar to Loevinger’s (7) index of homogeneity and to Menzel’s (9) co- 
efficient, of scalability, and is proposed only because it is easier to compute 
in the present circumstance. 

. According to Guttman (4), a set of items should meet several criteria 
in order to be considered "scalable," or homogeneous. These criteria were 
apparently generated from anxiety about the chance rep [Festinger (1)]. 
With the exception of the requirement about а random pattern of errors, each 
criterion is related to the problem of avoiding spurious results and achieving 
homogeneity. It seems natural to replace them by a single criterion concern- 
ing Т: I should be .50 or more for scalability. This criterion appears to give 
roughly comparable results to the many criteria used heretofore, and will be 
helpful to those who desire to create a dichotomy of scales vs. nonscales. To 
this author, it seems preferable to evaluate а scale in terms of an index of 
consistency that varies along a continuum rather than in terms of an arbitrary 
dichotomy. 

. Scoring. Any scoring method may be used with the pret лаа 
Since none of the previous steps depends on the scale scores. The pu я 
of rep refers to the reproduction of ап individual's item тро TO 2a 
knowledge of his scale score. The measure of rep is a measure of t 3 WU 
or this reproduction when the scores are 50 assigned that the Ae ES scoring 
15 minimized. It follows that the logically consist Вр 


епф шей | i 
Е scale 

1 водепів is to compare their responses with the scale types. Each 

Ype or perfect response pattern is assigned number, 


usually the number of 
i indivi 4 опзе 
Positive responses in the response pattern. Then each individuals ober 
pattern is compared with the scale types. An individual s score 5 а 
assigned to the scale type that his responses match with the X EU xe 
d en the scale is perfectly reproducible, the d а e 
Н termined by this scale-type-comparison process 18 equiva Т зз 
ü Positive responses that he made. Нарр! et E 
d ] that the number of positive responses is very highly ¢ 
a €-type-comparison score, when th 
bad d aniio odd DE is Although the issue 15 80 


Scoring meth iz pod ons 
od of counting positive resp Е ft 
What academic for small numbers of items, the savings O 


a : А 
Те sizable for large numbers of items. 


Non-dichotomous Items | 
analysis wit! 


In order t b 

o use the presen ө 

m i ; ator mus 
d re than two response alternatives, sug 

Jw) Several possibilities exist. He 

9ns, perhaps depending on the co 


may u$ 
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The splits might be determined in part by the popularities of ie ae 
It would be possible to use some method such as that propos А y МТ 
(5) for attempting to choose the dichotomies that maximize rep. s ез et rs 
each respondent could be given a provisional scale score, and eac! in er 
dichotomy of each item could be correlated with this score; the dichoto 
with highest correlations or covariances would be chosen. | ad 
An alternative procedure is to use all possible dichotomies. That 15, : 
n-alternative item is replaced by (n — 1) dichotomous items, or аи i rn 
For this procedure, it is necessary to adjust the value of k in the TRUM 
Wherever k appears as a factor, in conjunction with N, it remains the num. E 
of original non-dichotomous items, but in the limits of summation, k becom E 
the total number of pseudo-items. The pseudo-item trick will provide І 
crude approximation to the rep of a set of non-dichotomous items. Tan 
approximation is always slightly too low, but the value of Z should be more 
accurate since the same bias exists in both Rep, (or Бер») and Rep, . 
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NOTE ON CARROLL’S ANALYTIC SIMPLE STRUCTURE 
Henry F. KAISER 

UNIVERSITY OF CALIFORNIA 

trix for Carroll's analytic 


lves successively finding 
etric matrices. 


nsformation ma 
procedure invo 
d vector of symm 


A method for computi 

P puting the {га 

simple structure (1) is P ented, The 
e smallest latent root and associate 


applyi f carrying out the iterations required in 
b шн. Carroll’s analytic criterion for oblique simple structure (1) is 
вер oped. Since the trial and error procedure Carroll recommends occurs 
^» y in the computational stage of the solution, the general treatment of . 
e problem will not be repeated. The notation used here is that of the 
original paper. 
Carroll's criterion is that 


A mathematical method o 


pct E n 
ў = рж Viia (D 
pel qepil iat 
r è n E 2j = 2 
A (È amm) (2 р @ 
j=l \т”1 mal 


be "NC . 
a minimum, under the é conditions 


I 7 vad, = LG 


m=1 


(8) 


he elements of the 


matically varying t | 
minimum. To this 


"M He reaches a solution by syste д 
У асси matrix A until the value of f attains 2 
nd, he is able to write 
4) 
- МАМ ( 
Where Se 9 Ro 
ГА 
м: = IN , M. ay ttt Ne › Mugen 777 S , NN: › (5) 
sam 0 Rakes ur a ncs 


15 а row vector with s(s + 1)/2 variable elements, consisting of pod 
umn т of the transformation matrix А; 
т complicated 


e terms of the elements of col 

ere AM, is a column vector whose elements are rathe n F 

шша of the elements of the arbitrary reference factor matrix ar 

he elements of the columns other than 7 in A; and where у. із that part о 
89 
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f which varies as a function of the elements of column x of A. Thus, in any 
one iteration f. must be minimized with respect to A, : 
The s(s + 1)/2 constant elements of AM, are rewritten as 


АМ» = {bn Ho TEE AD s 2035, ++: ‚ 2, , 2b, Й 


(6) 
171 42b, , S 200], 
and the symmetric matrix В is defined as 
bi bie P b bis 
bey baz Е E be, 
B= , (b, ар һы). (7) 


b, bie Ы u bis 

[Note that the elemen 

of АМ» ; while the off-diagonal elements—wi 

scripts—are one-half the value of the final s(s — 1 
From (5) and (6), equation (4) becomes: 


a a-1 а 
je Xd Y WALL. (8) 
т=1 1 


т=1 k=m+ 


But the Xn, are constrained by (3), so that to minimize f, let 


20 = f, — ug, 
а a-1 a а 
SON VES bus, — (з Anè — 1), 
m=1 m=1 kem] m=1 
where y is a Lagrangian multiplier, 


To find critical values of An- set the partial derivatiy 


€ of (9), with respect 
to any one of the s variables Amz , equal to zero: 


90/0. = (Bam — Lol y DVL est (10) 
kel 
kám 


If (10) is multiplied by Amz , and then summed over m, 


m=1 k=m+ 


г 2—1 в s 
25 eda Э Usu, = 7... a1) 
mel 1 m=1 
Upon using (3) and (8), 


= jf. (12) 
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Thus writi 
| writing out (10) explicitly for m = 1, 2, +++ , s gives 
" У 3 ? 3 
B 13. Ы... е b 0 
1s^sz = 
Bad + 
21^1z (baz = 7%. + +? + [7 m =0 


bad + + + (b. — = 
| iz ..№, „Б. (b.. ЛӘ) sz 
Since these are h g inear e i 

i omo: zeneous ine: quations, the determinant of the 


(bis =» fe) bis "Mc bis 


bar (бә — 7.) И [m 
. = 0. (14) 


ba bia (6. = f2 


No А Е 
of which id is obviously the characteris B, any latent root 
smallest een the determinant vanish. In order to minimize f. , the 
With this fo root is taken. The elements of the latent vector associated 
ER ex are obviously the desired solutions for Amz - 
ample, in Carroll’s first iteration for Thurstone’s box problem 


(1, p.34), В becomes 


tic equation of 


3.931 —.037 .064 


—.037 1.708 44 |. 


.064 44 1.208 


tforward computational scheme for de- 
ent vector (2; р.357-8), 


Employi A ; 
ploying Tintner's straigh 
d its associated lat 


termini 
hing a smallest latent root ап 
Aie Cc .025 (.000), 


Aa. = —.287 (= 292), 
№. = -957 (.956), 


Th 7. = 1218 (1.219). 

er ы | 

апа S in parentheses are those obtained by Carroll by means of a trial 
rror procedure. 
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BOOK REVIEWS 
d Edition. New York, John Wiley and 


Quinn М т 
oe MURS Psychological Statistics, Secon 
‚ Inc., 1955, pp. vi + 408. 


The ui 

the first e ine par of MeNemar's well-known text has the major virtues and faults of 

unchanged, og : hough the revisions are considerable, much of the original text remains 

The eei ractically all of the revisions represent improvements. 

same field—is et e tue of this book—not shared, unfortunately, with some others in the 

EE зи relatively complete freedom from major errors. The instructor who uses 

aro'incorrect, Tt o spend much time explaining wherein and why the author's discussions 
‚ Its primary fault is still its unevenness of exposition: careful, detailed, and 


lucid o 
n many topi А 
y topies, condensed оп several others to the'point where students are unable to 
hor could not quite bring himself to 


acquire any г 
omit беъ a: understanding. Apparently the aut 
fully. If cn бе of topics which he evidently did not feel he had space to develop 
lent, but if he $ rar is willing to skip over these topics, he will find McNemar’s text excel- 
the text frequ o he must teach them thoroughly he will find it necessary to supplement 
The Ош їп his lectures and additional a gnments. 
als of the саті non improvement in the revised edition is а rearrangement of the materi- 
the t-test үш with considerable re-writing at various points. The discussion of 
of one variabl e that of correlation, so that all the materials on the sampling theory 
expanded Жү ollow consecutively; the treatment of the binomial distribution has been 
hough seven hb more fully as a starting point for the ideas of statistical inference. 
Nas increased Т apters on the one-variable case now replace six, the total number of pages 
only from 108 to 114; the reviewer still doubts that this treatment is sufficient 


or stude 
т zi : 
quM Mb their first course in statistics. 
(still too brief) m RU improvement is in the treatm 
on testing variances for homogeneity has been 


complex 

X analysi p 

A am sis of variance has been improved considerably. 
page chapter on distribution-free methods presents the sign test, the median 


test, M, 

ood" 
More like x test of C correlated sets, and the Mann-Whitney U-test. This chapter seems 
he result of an afterthought than a serious effort to discuss à now important area 


of mo е 
Te statistical inference. 
tribution ea oe chapter entitled “Distribution Curves” describes only the normal dis- 
it, are бо standard scores. The binomial distribution, and the normal approximation to 
A few cred in the following chapter. The Poisson distribution is not discussed. 
error of the i inor errors and inconsistencies still remain. Thu: 00 the standard 
7 is the popu] ean of a sample from a finite population is given as 
Instead of ation standard deviation, N the populatio and n the sam) 
given as с, аз о /N —n/ N/n(N — 1). Also on page 133 the standard error 0 
Пее 1 Мт — f, omitting the factor wW- DIN -2 —DAN = 2): Similarly on рр: 138-9 
again with ion 1 — 2.2/0, js defined as the proportion of the i determined y 
of multi; au the factor (N — 1)/(N — 2» though the correct i à 
he Шш е correlation on page 186. The so-called “s » multiple correlation is in fact 
ecomes ee of the coefficient of non-di in the two-variable case this 
Correlati -1—ü0-UN-0D AN — 2. Me “shrunken” multiple 
i on the unbiased estimate of the multiple correlation in the population; if this is 80, 
r has never seen 3 


T'as det 

: n : ‹ 

18 75 an = yE ove is the unbiased estimate of p. The reviewe seen а pue that 
à nbiased estimate, and in fact it is probably 20 te it in the form, 


ent of analysis of variance. A chapter 
added, and the chapter on 


biased estimate of o; ; 


t. For if we wr 
and 52 is an un 
of its numerator 


r2 
= 1 52 
the unbias а , 52., is an unbiased estimate of су. » з Т 
аа q ей estimate of a ratio is seldom the quotient cf unbiased estimates 
f a square root is seldom the square root of 


ienomi 
minator, and the unbiased estimate о 
93 


94 PSYCHOMETRIKA 


the unbiased estimate of the number. Aside from these and a few other liberties with such 
ratios as N/(N — 1) and (N — 1)/(N — 2), however, there are few errors. 


The formula for the large-sample standard error of (ris — ri3) has been replaced in the 
revised edition by Hotelling's ¢-test for this difference. 

To the present reviewer, the author's deliberate lack of em 
methods appears to be a deficiency, but 
many other authors agree with McNemar. 


In summary, those instructors who liked the first edition of McNemar’s book will like 
the second edition better; those who did not like the first edition will probably have the 


same objections (though somewhat weakened) to the second. This reviewer, it may be added, 
is one of the former. 


phasis on efficient computing 
this is admittedly a matter of opinion wherein 


University of Tennessee Edward E. Cureton 


R. L. THORNDIKE and ELIZABETH HAGEN, 


Measurement and Evaluation in Psychology and 
Education. New York: Wiley, 1955, : 


рр. vii 4- 575, 


established uni: 


widowed, or divorced) ; zy 


with examples of E 

The chapter on 
tendency, variability, 
interpretations of Standard deviation m 


A : r 
T ltem construction аге given, togetbe 


sophistication the authors have chosen, 
The sixth chapter deals with test desiderata: validity, reliability, and practicality, and 
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culminates in a Schedule for E i 
ies Evaluating a Test. The discussion of reliability i 1 
ae емо of validity differs from that of the recent pie. 
validity" ү uis ти particular gain, it seems to me. Furthermore, the term “construct 
seis with a meaning different from that of the APA committee. 
теи а types of norms—age, grade, percentile, and standard-score—are 
КОШЫ oft dh ed In the discussion of standard scores the authors seem to go 
kem. "e E ris rack, and are actually perpetuating the prevalent fallacy that there is 
een t. dee cem of subtracting the mean and dividing by the standard deviation. 
м ен a an a ecause the units of a score system based on percentiles are so clearly 
the ud m : 4 to look for some other unit that does have the same meaning throughout 
Аң е E з values. Standard-score scales have been developed to serve this purpose.” 
pre a кыра in which Mary gets а score 1.50 above the mean on test А and Johnny, 
idu koe png class, gets the same standard score on test B. “Thus, we may say that 
flee aoa on test A as Johnny did on test В...” (р. 166), To aid in interpreting the 
сенй ia pe represented by a standard score" the reader is referred to a table of 
di Lom enia ents ina normal distribution. This puts the authors in the peculiar position 
Asai adi he section on standard scores with a statement on the inadequacy of percentile 
Й then midway in the discussion using them to interpret the supposedly superior 


Standard scores. 
Eus phe nly several paragraphs beyond this that they take up the matter of normalized 
equal ris Th In a summary, however, they say that standard scores аге“... presumably 
menn of йыры basic unit is the number of standard deviation units above or below the 
the group” (p. 168). 
каана seem to be too easy 
audience км from this presentation. I should think it entir 
an Кы ис the book is directed to understand the ration2: 
only line normal distribution of ability, and that non-normalize 
some ki d transformations on raw scores, better than raw 80 
tests euh reference distribution or unless i esired to compare performance 
Adan have the same form of distribution. | ’ 1 
the пе is same chapter has an excellent discussion of profile interpretation, 
cessity for taking into account the reliabilities of difference scores. 

ESSE. 8 through 15 cover the topics which form the heart of the usual tests and 
which pe ments course: sources of information about tests, the various kinds of traits for 
gener: tpl have been designed, and kinds of measuring instruments in current use. The 
RE approach here has been to try to set forth principles governing various measurement 
ques so as to give the reader & background for evaluation. Illustrative tests are dis- 

summarized. 


cussed briefly, and findings of some of the research studies in relevant areas are т 
The last five chapters school testing program, marking and 
diagnosis and 


reporti deal with planning а Y 
ү rting, educational and vocational guidance, personnel selection, and E 
erapy, In the chapter on Marking and Reporting, Thorndike and Hagen take the position 


a false picture of the virtues of 
ely within the grasp of the 
le for using a unit based on 
d standard scores, being 
ores unless there is 
across 


for the naive reader to get 


are no 
t is d 
emphasizing 


Fd Course marks can be only a relative appraisal, with respect to some reference group. 
һа: ey ignore the alternative of assigning marks based ол the extent to which the m 
d achieved the operationally defined objectives . Sufficient pd zi 
its made in this direction, certainly, to make it a f and, for me 
st, a preferable one. 
cone ше are four appendices, the first two 
SHE n coefficient). The third is а listing and eva 
exl ely used tests, and the fourth is a list of seven pro! 
lescription of the kinds of services 


of which are computati 
luative description of во: 
minent test distributing agenc 


they offer. 
John E. M ilholland 


University of Michigan 
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Zditors, John 
Decision Processes, Тнвли, R. M., Coons, С. H., AND Davis, R. L., Editors 4 
Wiley and Sons, New York, 1954, viii + 332, $5.00. 


ci i i j sisting 
Decision Processes while nominally a book, is in fact a one-issue p E "€ 

of аа mathematical and experimental papers on statistical decision theory, 
o x m 


The Temaining articles are grouped in f. 
learning theory, theory 


impossible to discuss th 
found particularly 
are represented. 


L. A. Goodman's Paper "On Methods of 
Against Nature,” and the “N 


7-3 пег 
ote on Some Proposed Decision Criteria" by Roy Rad 
and Jacob Marschak are all со i i 


our sections: individual and social "€ 
and applications of utility, and experimental studies. Since 1 er 
em all in detail, attention will be restricted to those the de 
satisfying or stimulating; as it happens all four divisions of the bo! 


ich 
; Bayes, and opeland. Radner and Marschak present an example whi 
Suggests that both the Hurwi 


кы the 
urwicz generalization of the Wald minimax criterion and 
Savage minimax regret criteri i 

raises similar doubts. The Н, 


three others as desirable; he shows that the class во 

characterization of this cl 

an unsolved problem. 
The first paper of part II, 


018 
defined is non-empty. Finding a pu^ 
9f any member of the class, rem 


" 
“A Formal Strueture for Multiple-Choice Situation? 


` of the B 
mathematical structure of the Bush-Mosteller stoc! 
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by R. R. B 
. R. Bush, F. ick M 
rederick Mosteller, and G. L. Thompson, is a welcome concise statement. 
hastic learning model. As is well 


known, the 
› model са in vi 
can be stated in very general terms, but most of the results and appli 


cations assume li 
SUN gm oe a coh A major and controversial part of the paper is an att 
for this орно tha ining of classes and condition,” to give a more respectable basi А 
төшен tek Шо шо ae Ше intriguing observation that it works. Roughly, this vondition 
rcgis aie les de! yield the same results whether or not two alternatives with the 
PIN DE: cte ein are combined. At first glance this seems to have the 
be аа а ive necessity as, say, the requirement that the laws of physics shall 
ita exciting, Caref ү position of the observer; to the extent it has this status and necessity 
Оша: fo di eful inquiry, however, suggests otherwise, for the probabilities relatin 
the model oo wa are under the arbitrary control of the experimenter; bodie 
that Yid e а for any possible combining of classes. It appears to the mes 
justification ja d to be considered intuitively necessary, and thus is not really з 
linear model HE е linearity assumption. Still a persuasive justification is needed, for the 
"Individual B : an impressive collection of data. An example of such data is presented in 
Associates ehavior in Uncertain Situations: ‘An Interpretation in Terms of Statistical 
ee tl gun i by W. K. Estes. | 
a іна on utility, includes two papers on the existence 
eresting but mathematically the most difficult in the book. The first, “Repre- 


Sentati 
on of a Р s 2 

reference Ordering by а Numerical Function" by Gerard Debreu, is con- 
ordered set which are sufficient to insure 


Cerne f 

the Фе topological conditions on a weakly 

bility and — a utility function. If certain sets are closed, he shows that either separa- 
combining is о ог perfect separability are sufficient. No algebra of probability- 
Tesults are obt, T umed as in the von Neumann and Morgenstern theory, but no unique 
of dropping gern In “Multidimensional Utilities" Melvin Hausner examines the effect 
Let ApB den ie Archimedean axiom from the von Neumann and Morgenstern axioms. 
Preferred to T. e a probability combination of А and B; the axiom requires that if A is 
to the axiom ig and В to C, then ApC and B are indifferent for some p. The possible objection 
obtains the | s seen when one lets A = five cents, В = two cents, and С = death. Hausner 
elegant results that any non-Archimedean “mixture” space satisfying the other 


von Neum. 
that ann and Morgenstern axioms can be imbedded in an ordered vector space; and 
lexicographically ordered in some basis. Some interesting 


gested by R. M. Thrall in “Application of Multidimen- 


of utility functions; these 


initiates a fascinati of Organization and Information” by Jacob Marschak 
scinating normative study of decision-making by communicating “teams,” 
with identical individual and group utility functions. 

Cost, and tak К nformation over à communication network at some 
ога ul = Actions based on a decision rule. Three classes of problems are considered 
ШЕН айр which completes all observ e making any decisions. 1) Procedural: 
tion trans work and cost of communication, to select the best rules for governing informa- 
mission and actions. 2) Network: given rules and a cost function over networks, 

ect the best procedural- 


to sel 
ey the best communication network. 3) Constitut с 
pair. Several simple special cases аге solved, but as Davis notes (p. 13): “The 
ed even for these simple cases show for one thing 
n of the theory would be, while on 


relati : 

i d Paus manipulations requir 

the ОГА le further development and 

the i ees they serve to emphasize h 
Ta th vá of this formalization.” 

in the e final experimental section, two of the four papers 

game-theory sense; both emphasize that psychologic 


simplificatio 
М lysis at all without 


ow difficult would be 


deal with co: 
al rather t 


any ana 


alition formation 
han “objective” 
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“Tendencies Toward Group беш 
; Leon Festinger, and Douglas M 
= eories of group behavior to predict that those who E 
i relevant to the conflict of interest involved tend а d 
+ The confirming experiment was based on a symme б. 
ays a stooge who, in one variation, appeared to be 


ability in Competitive Bargaining,” Paul Hoffman 


in 
ting. Several n-person games (n = 4, 5, 7) were run In 
characteristic function form, i.e, payments Were stated for each possible coalition. 
each case subjects bargained for 10 mi 


рреаг to be: contrary to Me 
; the Shapley value tended 0 ry 

› to the experimental payments; no satisfactory 
method was devised k 


E 


rom other apparently less impress 
те not great, the experiment, prob, 


ing many 


ive coalitions. While the prospects 9 
ably should be replicated чо bs 
more subjects, At that timo data coul ion. 
each run as to their perceptions of relative coalit! » 


do not expect these to be the same as the "ration? 
ive characteristic functi 


R. Duncan Luce 
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OPTIMAL TEST LENGTH FOR MAXIMUM ABSOLUTE 
PREDICTION* 


PauL Horst AND CHARLOTTE MacEwan 
UNIVERSITY OF WASHINGTON 


_ The concepts of multiple differential prediction and multiple absolute 
prediction are developed in earlier papers (2, 3). The problem of determining 
the optimal distribution of testing time for multiple di erential prediction has 
been previously considered (4). This paper develops an analogous procede 
for multiple absolute prediction. A numerical i illustrating the proce- 
dure is presented. The mathematical rationale underlying the procedure is 


given. 


I. The Problem 


A technique was presented in (3) for selecting from a large number of 
predictor variables the subset of specified size which would have the highest 
absolute prediction efficiency for a given set of criterion variables. In (2), 
an analogous procedure was developed for selecting the subset which would 
most efficiently predict the multiple criteria differentially. In each of these 
cases, efficiency was defined in terms of the accuracy of prediction. In (2), 
differential prediction efficiency was defined in terms of the accuracy with 
which differences between all possible pairs of criterion measures could be 
predicted. An appropriate index of the prediction efficiency of a selected 
battery was shown to be the difference between the average variance of the 
predicted criteria and their average covariance. This index was designated 
by ф; the larger the value of ¢, the greater the differential prediction efficiency 
of the battery. 

In the case of multiple absolute рге 
terms of the accuracy with which all the cri 
of the extent to which the selected battery 
The index of absolute prediction efficiency of the selected battery was taken 
as the sum of the variances of the predicted criteria, regardless of their 


covariance; the larger this sum, designated by № the greater the absolute 
prediction efficiency of the battery. 


diction (3), efficiency was defined in 
teria could be predicted, regardless 
would differentiate among them. 


д 5 versity, 
*This rese: was carried out under Contract. Nonr-477(08) between the Universi 
of ARS arch Me Office of Naval Research. The computations S gani бш ру 
Robert Dear and Donald Mills. Much credit is due the VS E t] CES 81 пед ion 
of both computational and editorial activities was prov? ed by William d 
of these able contributors we are deeply grateful. 
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For both types of multiple prediction it was assumed that intercorre- 
lations between the potential predictors were available, as well as satisfactory 
estimates of the correlations between each criterion variable and the potential 
predictors. It was further assumed that predictors and criteria were in 
standard measures, and that the predicted criteria were the least square 
estimates. Ав was pointed out in (3), the essential difference between multiple 
absolute and multiple differential prediction is that the respective sets of 
predictors selected will differ. The two sets may show varying degrees of 
overlap; the extent of overlap depends upon the degree of correlation among 
the criterion variables and upon the original group of potential predictors 
from which the two subsets were selected. 

Both methods referred to tacitly assume that all potential predictors 
take the same amount of administration time, so that all subsets of the same 
size would take equal administration time. This will not usually be the case. 
The problem may be approached in a more general way by Starting with а 
given battery of predictor variables and determining how, for a specified 
over-all testing time, the individual test time-limits should be altered in order 
to maximize the index of absolute prediction efficiency. 


prediction involving multiple criteria. In this article the procedure developed 
in (1) is extended to the case of multiple absolute prediction. 


exact solution was indicated. 


In the present paper, as in the cases previously presented, it is assumed 
that intercorrelation, validity, and reliability data are &vailable for predictors 
of arbitrary length. Testing time is taken to be the time actually allotted the 
examinee for taking the test. Any alteration in testing time implies a corres- 
ponding alteration in the number of items. Consequently the terms "testing 
time" and “test length" are used synonymously. 

The method will first be described and illustrated by a numerica] example. 
Following this, the mathematical rationale will be presented. 
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II. Numerical Example 


The data used for this example are the same as those presented in (4). 
The predictor variables are: 


(1) Guilford-Zimmerman Aptitude Survey Part I, Verbal Comprehension 
Р (2) Guilford-Zimmerman Aptitude Survey Part III, Numerical Opera- 
ions 

(3) Guilford-Zimmerman Aptitude Survey Part VII, Mechanical 
Knowledge 

(4) A. C. E. Psychological Examination, Quantitative Reasoning 

(5) A. C. E. Psychological Examination, Linguistic Reasoning 

(6) Cooperative English Test (Form OM), Usage 


The matrix of test intercorrelations with reliabilities in the diagonal is 
given in Table 1. The criterion variables are grade point averages in each of 
ten college areas. The matrix of validity coefficients is given in Table 2. 

It is evident that the correlations of variable 3 with the criterion are 
all small, four of them being negative. In general the chief justification for 
including such a variable would be that it might serve as a suppressor variable, 
i.e., a variable which suppresses invalid systematic variance in the predictor 
variables. 

The original test lengths may be seen in row 1 of Table 3. The over-all 
testing time for the tests of arbitrary length is 142 minutes. We assume that 
this time is to be cut in half so that the over-all testing time is 71 minutes. 
The problem is to determine the time to be allotted to each test so as to 
maximize the index of absolute prediction efficiency. 

The traditional assumptions are used here as in (1) with respect to the 
effect of test length on correlation, and will not be repeated. As in (4), the 
method for solving for the new test lengths involves a series of successive 
approximations. The computational procedure to be described consists of 
the same sequence of operations as those given in (4), the difference being 
that in the current procedure the matrix of validity coefficients 1s used, 
whereas in (4) the deviation form of these coefficients was required. 

1. The first computational step is the calculation of the elements for 
a diagonal matrix A, seen in Table 3, row 3, labelled 1’A. Row 1 of this table 
gives the original lengths of tests; the elements in row 2 were obtained by 


subtracting the reliability of the indicated test from unity. The elements of 
in row 1 by the corres- 


A in row 3 are obtained by multiplying the element seve 
ponding element in row 2. Thus for the first element we have A, = 25(.080) = 


2.000. 
2. A first approximation is now required for the altered test lengths. 
We assume the new test lengths to be proportional to the original test lengths. 
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TABLE 1 


В Matrix of Predictor Intercorrelations with 
Reliabilities Substituted for Unities in the Diagonal: 


Кер. 
І 2 3 5 E 
St Баа IS. Rl 3 8. —E———— 
21 “920 .159 „150 .281 „163 .515 2.790 


G 
G-Z III -159 .920  .003 .369 -292  .2h3 1.986 
G-Z VII -152  .003  .920 .200 -142 -.150 1.267 

-281  .369  .200 .820 -5h9  .N26 2.645 
A.C.E.-L  .763  .292  .1h2 549 -830 .628 3.20% 
English 2515-243 -.150  .heó .628 860 2.522 


2.790 1.986 1.267 2.605 3.204 2.522 14,414 


Moun коюн 
> 
СЯ 
[5] 
Li 
o 


TABLE 2 
The г. Matrix of Validity Coefficients 


1 2 3 2 
G-Z I G-Z III C-ZVII ACE-Q ACE-L En lish b] 


1 Anthropology  .370  .177 -091 .29} „зу -357 1.630 
2 Chemistry 317 „270 .016 .309 1364 :399 1.679 
3 Economics -339 .211 — .008 .241 1334 -323 1.156 
4 English -526 .2h7 -.075 260 “488 520 1.972 
2 Foreign Lang. .295 .287 -.156 .200 .232 .h26 1.284 
6 Geology +184 „що +094 .170  .229 214 1.031 
T: History ЖЕЛ ИУ! -.001 .182 37 +336 1.438 
8 Mathematics  .287 1348 - 088 .350 T 

9 


Psychology „Mio „170 €096 28s WB xn l5 
10 Zoolo .336  .216 о 


TABLE 3 
Computation of lA YD апа l'a 
1 Y 


First approximation: Dy, = фі Da 


2 VD, -080  ,080 .оёо .180 -170 .1h0 
gina YA 2.000  .720 2.400 4.140 2,550 5.600 17.10 
1 Dy7.5(108) 12.5 — 1.5 15.0 11.5 7.5 20.0 71.0 71.0 
5 P Dp -1 080 „222  .067  .o8q -133  .050 

2 ge 1 -161  .360 
6 YAD, 160 лб ш 360. .339 „2% 1.460 


- lI 
OOOO 
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"Therefore, as a first approximation to the new test lengths we take one half 
the original test lengths. Row 4 in Table 3 is obtained as one half of row 1. 
: 3. Caleulate the reciprocal for each of the D, elements. These are shown 
in row 5 of Table 3. 

| 4. Multiply each A value in row 3 of Table 3 by the corresponding value 
in row 5. The products are entered in row 6 of Table 3. For example, the first 
element is .160 — 2.000(.080). 

5. Next, the elements calculated in step 4 are added to the corresponding 
diagonal elements of Table 1, and the table is copied into the upper left 
quadrant of Table 4. The first diagonal element is 1.080 — .160 + .920. 
Note that the elements below the diagonal are not copied in. The upper right 
section of Table 4 is the matrix r. , the transpose of Table 2. 

6. We next calculate a matrix L, by premultiplying the matrix 7, by the 
inverse of the matrix in the upper left quadrant of Table 4. The computations 
of the forward solution are given in the two lower quadrants of Table 4 and 
in Table 5. The back solution is given in Table 6, in which the transpose of 
the matrix L, appears in the upper left corner. The procedure for multiplying 
a matrix by the inverse of a symmetric matrix is outlined in (5). 

7. The second approximation to the new test lengths is computed in the 
lower section of Table 6 as follows: 

Row a consists of the sums of squares of the column elements of the Li 
matrix. For example, the first element of row a, namely, .441, is the sum of 
squares of the first ten elements in column 1 of Table 6. Row b is copied from 
row 3 of Table 3. Row c consists of products of corresponding elements in 


rows a and b. For example, for the first element, 882 = .441 (2.000). 


Row dis obtained by taking the square root of the corresponding element 
ck upon the computation 


in row c. For example, -939 = V .882. Row e, a che 
of row d, consists of the squares of corresponding elements in row d. Row e, 


then, should be the same as row c. The value computed to the right of row 
e, and labelled s, 5 is obtained by dividing the new over-all testing time, 71 
minutes, by 3.930, the sum of elements in row d. Thus the value of sı is 18.060. 
Row f is obtained by multiplying each element of row d, including the 
summation element, by sı . For example, the first element obtained is 16.964 — 
to the new test lengths 

1 the new over-all testing time. 


.939(18.066). This row gives the second approximation 
tin row b by the corresponding 


and its sum should equa 
Row g is obtained by dividing each element 
element in row f. For example, the first element is .118 = 2.000/ 16.964. — 
ments in row g to the corresponding’ 
le 1. For example, the first element 


Row h is obtained by adding the ele 
reliabilities as given in the diagonal of Tab! 
in row h is .118 + .920 = 1.038. 4 

ix i by repeating steps 5 and 6, and using 

8. A new L, matrix 1s computed by гер К А а 


the elements of row h of Table 6 in the diagonal positions o! A 
matrix may be seen in transposed form in Table 7, rows 1 throug i 


-1 -1 
Computation of Lc (RHA Dy ) Bi 
Recip i 1A 2А ЗА XA 5A 6A 18 2B 3B 4B 5B 6B TB BB 9B 10B [d 5 
1А 1.0 -159 .152 2l HE E Eo MEM 


TABLE 4 


Forward Solution 


1 .163 .515 -370 .317 .339 .526 .295 .1Uk -3719 .287 . .336 6.123 
2А 1.080 .003 .369 .292 .2h3 .177 .?]b „211 247 -287 .1h0 .169 .3%8 .170 .216 1.385 
3^ 1.081 .200 .1h2 -.150 .091 016 1008 -.075 -.156 .09% -.001 -.088 .096 .031 1.444 
bA 1.180 .5h9 .h26 .29} .309 211 265 +200 „170 .182 .350 .285 .318 5.616 
5A 1.169 .628 .3%1 .36h  .33& 488 -232 .229 .373 .336 .409 .345 6.994 
бА 1.140 .357 .399 .323 52h .h26 ур -336 .hol .h03 .351 6.536 
C 2.950 2.146 1.428 3.005 3.543 2.802 1.630 1.679 1.156 1.972 1.28% 1031 1.438 1.634 1.803 1.597 31.398 
-926 1B 1.080 .159 .152 .281 -763 .515 .370 .317 .339 .526 -295 .184 „379 .287 .Mho .336 6.423 
-946 2B 1.057 -.019 .328 .180 .167 .123 .227 161 -170 .2}} .113 .113 .306 .105 .167 3.441 
-9hh 3B 1.059 .166 .038 -.220 .0hl -.025 7.037 -.146 -.193 .070 -.052 -.123 .036 -.013  .600 
1.021 4B +979 .209 .275 .153 .160 .109 1095 078 -076 .057 .200 .132 .181 2.785 
1.953 5B -512 .162 .012 .055 .036 .06% -.03h .055 .071 .026 .040 .026 1.025 
1.441 6B -694 .123 14k  .086 .169 -195 .084 .088 .126 .13& .103 1.947 
TABLE 5 
Computation of І = (R4A Dy, ) го (continued) 


1 1A 2А ЗА bA 5A 6A 1B 2B 3B в 5B 6B TB — OB 98 10B с 5 
lA 1.000 тз тотыш ш ны DE DES 


бА 1.000 
„с OOUE EERE eee 
1B -1.000 -.147 -.141 -.260 -.707 -.477 -5.948 -5,948 


2B -1.000 .018 -.310 -.170 -.158 
3B -1.000 -.157 -.036 .208 
ав -1.000 -.295 -.281 
5B -1.000 -.316 
6B 


-.343 -.29h -.31% -.087 -.273 -.170 -.351 -.266 -.007 -.311 


-.116 -.215 -.152 -.161 -.231 
-.039 .024 „035 .138 .182 
-.156 -.163 -.111 -.097 -.080 
-.023 -.107 -.070 -.125 .066 


-107 -.107 -.289 
.066 .049 .116 
-078 -.058 -.20h 
-107 -.139 -.051 


-1.000 -.177 -.208 -.12% -.2lh -,281 -.121 -.127 -.182 - 


.099 -.158 
-034 .012 
.135 -.185 
.078 -.051 
-193 -.148 


-3.255 
- .566 
-2.843 
-2.002 
-2.806 


SIT 


VMIHLIKNOHOASd 


А 


oe i АЈиныыыс сиы eee ee 


ТАВІЕ 6 


Computation of Ly = (R+ а Y! =. Back Solution 


1 1A ФА 2B 3B 4B 5B 6B ec 

1 +235 .059  .059 116 -.033 JT] =ї .002 0 

2 .120 .147  .003 .092  .0hl .208 -1 -.001 0 
3 .203 .106 -.021 .067  .031 ‚12 -1 -.001 0 
4 -330 .108 -.091 .01% .08 244 -1 -.001 0 а 
5 «225 .196 -.125 .OhT -.155 „281 -1 .001 0 m 
6 .035  .070  .085 .02h .069 .121 -l -.001 0 m 
T .215 .072 -.025  -.007  .099 .127 -1 .001 0 S 
8 «127 .212 -.102 .155 -.007 .182 -1 .001 0 а 
9 .268 .0h3 .060 «076 .017 .193 -1 -.005 0 К 
10 .188__.090 -.00h .1h2  .00h — .158 -l .001 0 5 
a a Dury 1 151 .050 079 5 .з52 E Check ki 
b YA 2.000 .T720 2.400 4.140 2.550 5.600 Е 
ë dA Dn -882  .109  .120 +31 .115 1.971 E 
a Y (0 vat -939  .330 .346 .572 .339 1.40} 3.930 3.930 Е 
те. Я 
ў 1 Q 
e Y( LA, Y A) .882 .109 .120 зәт .115 1.971 8, = -—— COT S 
lk ES xD y ay” 3.930 = 
E , * LL 5 

f r -Y (Dr 5) 316.96. 5.962 6.251 10.334 6.12} 25.365 71.000 71.000 „р. 


g Y A Dp} .M8 121 384 .hoo0 .№16 .221 1.660 


2 -l 
h рр +1 AD 1.038 1.041 1.30} 1.220 1.246 1.081 6.930 6.930 


LIT 


SII 


TABLE 7 


Computation of І. = (R+ Ay ) го Back Solution 


-1 

ts ee 1А 2А ЗА TA 5A. 6A — ІВ 2B 3B LB 58 6B TB UB OB 108 chek — — 
1 -247 -061  .0h9 .110 -.0h0 186 -1 -.002 0 
2 -126 „153 .008 085 026 +227 -1 -.001 0 
3 .216 lll -.015 .062 .016 .135 -l -.000 0 
4 ~ 350 113 -.071 +008 „020 268 -1 -.001 о 
5 +218 — .203 -.099 .030 --1151 +307 -1 .001 0 
6 „Ohl .073 .от1 .026 .056 ‚127 -1 -.002: 0 
7 .235 076 -.019  -.006 072 .1h0 -l .001 о 
8 .125 .223 -.079 -139 -.01& +204 -1 001 о ч 
9 -288 — 046 „0520 073  .002 202 -1 .001 о a 
10 meee 299. , 11095: 55.001: — Зи -:007 116 . -l1 .002 о Е 

а а EL Li. E ст == ===— ML NN ЕЧ 
а V» 489 — .165  .032  .067 .03& ащ E Check S 
b 1'A 2.000 .720 2.400 1.1% 2.550 5.600 E 

1'® r А 3979 лә  .077 -277 .087 2.318 3.856 3.856 Я 
а 1°(D, 14 4) i .989 305 „277 -586 .295 1.522 3.954 
е Y(D;4) (0174) 2 978 9.077 217 .087 2.316 вр= —— T = Т. 17,956 
12155) “Plats "(рь Aj, 3,95 
1 (pr 4)*1 

f 1, = YA is, 17.758 6.195 !.97& 9.445 5.297 27.329 70.999 70.998 
Е Е l'an, -1 13 -116 „82 438/481 +205 1.835 


= 


, оа 
AD. S 71.033 1.036 1.402 1.258 1.311 1.065 7.105 7.105 


E ч 


EY 
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9. Step 8 is repeated in rows a through ћ in Table 7, where a third 
approximation to the new test lengths is seen in row f. 

Steps 5, 6, and 7 were again repeated to obtain a fourth approximation 
to the new test lengths. The values so obtained are shown, together with 
those of the preceding approximations, in Table 8. The vector of altered 
test lengths has not yet completely stabilized, but may be considered suf- 
ficiently so for all practical purposes. 

10. The index of absolute prediction efficiency, \, is computed as follows: 

и (а) To obtain the index A, corresponding to the first approximation to 
optimal test lengths, calculate the product of each value in the L, matrix by 
the corresponding value in the validity matrix seen in Table 2, and sum the 
products. Thus the value of ^, , seen as the first entry in the column to the 
far right in Table 8, is 2.053. 


TABLE 8 


Successive Approximations to ть for Ту = +0 = 2 


Value of for 
successive values 
Approx'n TAR 3 4 5 6 bj inL 
(.5) 1 Dg : 1 12.50 4.50 15.00 11.50 7.50 20.00 71.00 ш 2.053 
2 16.96 5.96 6.25 10.33 6.12 25.36 70.98 Lo 2.086 
3 17.76 6.20 4.97 9.44 5.30 27.33 71.00 L3 2.089 


4 17.90 6.30 4.61 9.10 4.81 28.28 71.00 &L 2.090 


(b) For X, , use the 2, matrix instead of 1л, and repeat the procedure 


described in (a). 

(c) To obtain any subsequent index, A; , 
the L, matrix, while following the procedure indicated in (a). 

The values of à show only a very small increase in this particular illus- 
tration. From the initial value of 2.053 to a value of 2.090 corresponding to 
the fourth approximation, the increase is .037 or less than two per cent, 
although some of the test lengths are altered to a considerable extent. In the 
case of differential prediction (4), the increase in the efficiency index, %, 
based on the same original data, was relatively larger, four per cent, for the 


fourth approximation to optimal test lengths. Р ТУ, 
Computations were also carried out with the original over-all testing time 
unchanged, and with the over-all testing time doubled. Three iterations were 
calculated for each of these conditions. The successive approximations to 
be seen in Tables 9 


optimal test length, with the corresponding А values, may nans 
: Р itions, also, the increase in А is small, 


and 10, respectively. Under these cond И x oe 
although the alteration of some of the test lengths is relatively great er a 
that found in the first illustration. For unchanged over-all testing time, the 
increase in А from the first to the third approximation 1s 1.5 per cent; for 


substitute the L; matrix for 
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TABLE 9 


Successive Approximations to 14, for т, = Т, 


о = 12 
Value of X for 


Successive Values 
rox'n 1 2 3 4 5 6 b in L 


Da: 1 25.00 9.00 30.00 23.00 15.00 0-00 112.0 "iy 2.203 
2 32.5% 10.00 10.45 21.42 18.66 48.93 182.00 L5 2.230 
3 32.87 9.70 8.21 20.21 21.61 49.40 142.00 1 2.234 


TABLE 10 
Successive Approximations to YA, for Ty = 2%, = 2(142) 


Value of X for 
Successive Values 


2 63.21 15.81 16.23 43.98 56.11 88.66 284,00 Ly 
3 64.22 13.98 12,21 43.99 67.65 81.96 284.01 


Ly 2.375 


if any, prediction efficiency may be expe 
of variation in relative test length. 


ш.м athematical Derivation / 


= the number of cases, 
n = the number of predictors. 
= the number of criteria. 
Z = an (M X n) matrix of test scores in а battery of altered length with 
the elements of Z of the form (z; — 2,)/(s,,/ MD). 

an (M X N) matrix of criterion scores whose elements are deviate 
scores of the form (w;; — 1;)/ (ov, VM). 

В = ап (в х N) matrix of regression coefficients for 
r = ап (n X n) matrix of intercorrelations of tests 
p = an (n X n) matrix of intercorrelations of tests of altered lengths. 


* 
11 


p 


Te 
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Te = an (n X N) matrix of validity coefficients for the tests of original 
lengths. 
pe = ап (n X N) matrix of validity coefficients for the tests of altered 
lengths. 
D, = an (n X n) diagonal matrix of original test lengths. 
Dy = an (n X n) diagonal matrix of altered test lengths. 
D. = рр; = the ratio of altered to original test lengths. 
Dn; = the (n X n) diagonal matrix of reliability coefficients for the tests of 


original lengths. 
The index of absolute prediction efficiency as defined in (3) is given by 
х = trG, 


Where C is the covariance matrix of predicted criterion scores in standard 
measure. Let 


ё = [I + (р. — рр, ,.]р;'. (1) 
Let 
є = (ZB — W). (2) 
We wish to minimize the trace of e'e with the constraining condition, 
1'D,1 = T, where T represents the new over-all testing time and 1 is а 
column vector of all unit elements. From (2), 
ee = B'Z'ZB — B'Z'W — W'ZB + W'W. (3) 
From the definitions above, 
Z'Z = р, (4) 
ZW = pe (5) 
From (3), (4), (5) i 
e'e = B'gB — B'p, — В + W'W. (6) 
Let 
y = tr de + M'DAM, (7) 
Where ) is а Lagrangian multiplier. From (6) and (7) 
dod (EIU — Bp, — ВР Е КЫ, (8) 
In (1) it is shown, for the case of a single criterion variable, that 
"T (9) 
t was 


i iteria. I 
а relationship readily seen to hold for the case of multiple criteria. 


also shown in (1) that (10) 


p = FU — D, + 0.000587, 
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where D, is defined as 7 — D,,, , a diagonal matrix of test unreliability 
coefficients. Substituting (9) and (10) in (8), 


y = tr [B'e — D, + D,D,D,')8^?B — prg — re — r8 "B 
] 


(1) 
TON + M'D,1. 
Let 
B = 31, (12) 
T= DIER (13) 
D,D, = A. (14) 


Then (11) becomes 
YR PR + ADIL Trac EN AD. — qs 


The unknowns on the right side of (15) are L, D, В 


М and à. Differentiatin 
(15) with respect, to row vectors of L’, and setting tl os ; 


his derivative equal to 


ð 
3b = (R+ADDL -r 0, 
or 


(R + Ару)г, =r, . (16) 


Differentiating (15) with respect to D, and setting this derivative equal 


to zero, 
oy А 
aD, = М — Dy. AD; = 0, (17) i 
where D;,. isa diagonal matrix who: =! i 
CEST are d se non-zero elements are the diagonal 
D, = (D, A) AV. (18) 
From (18) 
I'D = T (Dirr Ay? уух, (19) 
or, since we have the constraining condition, 1’D,1 = m 
ЕД 
м^ = I'(Di, A)". (20) 


Substituting (20) in (18), we obtain 


R, (Di. A) p 
D, = U (Dip A)T (21) 
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From equations (16) and (21), the formulas are derived for solving for 
D, by a series of successive approximations. From (16) we obtain 


L = (R + AD). (22) 
Let 
L: = (R + ADV) r. (23) 
where 
D.T 
D, = ТФА? (24) 
апа 
Dig, = Diem (25) 


FG Nl 


The first approximation to D, is indicated by (24). The second and all sub- 
sequent approximations to D, may be obtained by an iterative procedure 
based on (23) and (25). Thus, successive approximations to L; and Dsi 
may be computed until D, stabilizes satisfactorily. 

The regression vectors for the tests of optimal length will be given by 


Ві= pupae is 
From (9), (10), (13), and (14) | 
В = S^(R + ADV) re, 2 
and from (22) and (27), 
(28) 


B D, 
where L has been stabilized through successive approximations. ba 
Furthermore, it can be shown that the index of absolute prediction 
efficiency, A, as defined in (3) is given by 
A = tr D're (29) 
It may be that one or more elements of Ds, , as given а и Q3) 
approach zero as 7 increases. In this case it would be better to 
in the form 
L: = риҷривру? + A) Di?r. - (80) 
iir) g i i 
i ould 
Although the computation of successive L; matrices by Eres oye 
be computationally more laborious than with (23), 25 
resulting from one or more near vanishing elements of £^ - 
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The computational procedure presented in Section II is related to the 
mathematical derivation as follows: 

Table 1 is based on equation (13). 

Step 1 is based on equation (14). 

Step 2 is given by equation (24). 

Step 3 consists of calculating D3, from D,, . 

Step 4 consists of calculating AD;' . 

Step 5 consists of calculating the matrix within the parentheses in 
equation (23) for the case ? = 1. 

Step 6 consists of the computation of the matrix L, from equation (23). 

Step 7 consists of calculating D,, from equation (25) for the сазе? = 1. 

Step 8 consists of the computation of the matrix L, from equation (23). 

Step 9 consists of calculating D,, from equation (25). 

In general, successive approximations to L and D, are obtained by re- 
peating steps 6 and 7 for successive values of in equations (23) and (25). 

Step 10 follows the procedure indicated by equation (29) to obtain 
successive values of А. 
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A LEAST SQUARES SOLUTION FOR PAIRED COMPARISONS 
WITH INCOMPLETE DATA*t 


HAROLD GULLIKSEN 


PRINCETON UNIVERSITY 
AND 
EDUCATIONAL TESTING SERVICE 


_A precise and rapid procedure has been devised for dealing with a matrix 
of incomplete data in paired comparisons. This method should increase the 
general applicability of paired comparisons since experiments involving large 
numbers of stimuli may now be shortened to feasible experimental proportions. 


Also, we may now use sets of stimuli which cover а wide range, resulting in a 
considerable number of 100 per cent vs. 0 per cent judgments, and still give a 
h of the observations. 


precise solution depending equally on eac о 


In scaling by paired comparisons, many Cases arise where one does not 


have complete data. In any problem where the range of the set of stimuli 
is great in relation to the discriminal dispersion, there will be no usable data 
for the extreme comparisons. This is usually the case in dealing with con- 
struction of scales for various sensory areas, such as brightness, hue, pitch, 
or loudness. In other cases the experiment is made less laborious for the 


subject by not requiring all is interested in studies of 


comparisons. If one і 
value judgments (6) where composite objects are used, such as (a and b 
vs. c) or (a and b vs. c an 


d d), then it is also highly desirable not to include 
all possible comparisons. For example, 


it may be well to omit comparison 
(a and b vs. b) or the comparison (a and b vs. @ and c). Such situations might 
arise in studying preferences for various foods, gifts, 


individuals, activities, 
ог goals. Any of the types of situations indicated above may give rise to 
a matrix of incomplete data for which a re 


asonably precise solution is 
desired. A solution for paired comparisons wit 


h incomplete data for the 
case in which the correlations are equal and the ratios of the discriminal 
dispersions are known is presented here. The general usability of paired com- 
parisons, especially in fields such as sensation and value judgments, will be 


; А h Contract Хбопг- 
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greatly enhanced by a precise method for utilizing all the available data 
when dealing with an incomplete matrix. 

The procedure to be presented follows the general method for handling 
incomplete data outlined by Horst (2, pp. 419-430), Kempthorne (3), or 
Kendall (4). The least squares solution for paired comparisons, when corre- 
lations are equal and ratios of discriminal dispersions are known, has been 
indicated by Mosteller (5). The problem may be stated in the following 
manner for the case in which complete data are available: 

For the case of errorless data let St, S*(¢ = 1, --- ,n;j = 1, --- ,m) 
be the scale values of the stimuli. Define the distance D* = S* — S*. From 
the values of p;; (the experimental proportions of judgments i > j) we use 
a normal curve or some other assumption to determine values of D,;, which 
are taken as estimates of D* . 

The scaling problem is to determine values S; , S; (i = le, n 
j = 1, -+ , n) such that У). У. (Du – S; + Sj)! is а minimum. The 
subscripts 7 and j are alternative subscripts each designating the stimuli 
from 1 to n. 


ijy 


Least Squares Solution for Incomplete Data 


For the case in which only incomplete data are available we may write 
nij 
Q= 2; (Di; — S; + Sy, (1) 


using n;; to indicate that the summation is over the available data. The 
sealing problem is to determine S; and S; (i,j = 1, ... , n) such that Q is a 
minimum. It should be noted that since the matrix of Di; for complete data 
is skew symmetric the matrix of D,, for incomplete data will also be skew 
symmetric. 
To determine the unknown S values that will make Q a minimum, 

take the partial derivative of Q with respect to S; , giving 

199 7 nij nij 

238, a 22 O — 8+8) + У(-р,+ S, —8. 0 
Опе term represents the partial derivative for the row in which S, occurs 
in each cell, and the other term represents the partial derivative for the 


column of entries each of which contains S; . The first term is identical with 
the second so we may write 


1 aQ nij 
238, = 2 È (S: — 8i — D,). (3) 


If n; represents the number of observations in row (or column) 7, then simpli- 
fying and setting the partial derivative equal to zero gives 


n5 - Ès- È D=o G — 1, --- ,m). (4) 


| 
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This set of n equations may be expressed in matrix notation. Let 


S = the column vector of elements S; . 

1 — a column vector of 1's. 

D = the matrix with elements D;; wherever observations occur, zero for each missing 
observation, and zeroes in the principal diagonal. An illustrative D matrix is 
shown in Table 3 provided a zero is put in for each missing entry. 

M = a matrix constructed from D according to the following rules: Enter —1 in M 
for each cell entry in D where data exist. Enter 0 in all other off-diagonal cells. 
The entry in the diagonal cells in л; ће number of observations in row (or 
column) 7. Note that M is a symmetric matrix and that the sum of each row 


(and column) is zero, and hence M^? does not exist. 
Z = D1, a column vector, the sum of the rows of matrix D. 


Using this notation, the set of equations (4) may be written 
MS - Z. (5) 


_ Matrix M has no inverse, but, in general, its first minor will have an 
inverse. If we specify an origin by some method such as arbitrarily setting 
the first element of S equal to zero, a solution for the remaining elements of 
S is given by deleting the first element from Z and S, and the first row and 
column from M, giving 


MuSi = 21, (6) 


which has the solution 
х e 


It may be noted that for complete data M is an n x л matrix with 
n — 1 in the principal diagonal and — | in each off-diagonal cell. The inverse 
of M,, for, the complete data case is à matrix with 2/n in each principal 
diagonal cell and 1/n in each off-diagonal cell. In this case the solution given 
in (7) corresponds to the solution given by Mosteller (5, equation 10). 


Subsequent computations and the present analysis will be facilitated 


by the use of two other matrices: 


N = a diagonal matrix with reciprocal of (n; 


cell and zero in each off-diagonal cell. 1 Р 
1, = a matrix constructed from M by putting а zero wherever there is à zero 1n M 


and +1 in all other cells. 


+ 1) as the element in the ith diagonal 


Note that 
Ma Na аа (8) 


Using this notation (5) becomes 


5 = М8 – 18. (9) 
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For the matrix of complete data L becomes a matrix with unity in every 
cell and thus LS is a vector of constants. The origin of S сап be chosen so 
that LS — 0, whereupon (9) can be solved giving 


S = NZ, (10) 
which is also equivalent to the solution given by Mosteller (5, equation 10). 


Iterative Solution 


However, for the case of incomplete data where Мұ is difficult to 
compute, a solution may be found quickly by an iterative procedure. The 
procedure given here is an analytical analog of a graphical iterative procedure 
that was devised by Mrs. Gertrude Diederich, It was suggested by the pro- 
cedure outlined by Garner and Hake (1): Te corresponds essentially to taking 
their solution as a first approximation and then correcting it to obtain 
successively closer approximations. 

The iterative procedure proposed here for paired comparison is outlined 
below. We begin by taking a trial set of scale values. Since this corresponds 
to assuming a set of values for S we may say that T, designates this first 
Set of trial values. The discrepancies between the predicted values (MT) and 
the experimental values (Z) are found by taking differences (Z — MT.) = 
ME, . The average of the discrepancies for each scale value is then used to 
correct that value. This corresponds to taking the correction NM E, and 
computing the second set of trial values by setting T, = T, + NME, . 

Graphically this correction corresponds to changing each element of 
Т, by the average of all the discrepancies found for that element. When 
tried graphically, such a procedure gave convergence as far as detectable 
from a reasonably large graph in two or three trials, Although no analytical 
proof of convergence has yet been found, it seems intuitively reasonable 

tilizi tically would give a reasonable 
approximation to the solution. In any case, the discrepancies (ME,) would 
4 € of convergence could be readily 
detected. This process of computing T';,, = T, 4 NME, is continued until 
the discrepancies and the correction terms are negligible. 

А little matrix substitution shows that if T, = T, + NME, . then 
T, = T, + NI + LN)ME,. Thus, since the “two step” correction is very 
easy to compute, we find that T, — T, + NI + LN)ME, . In the compu- 
tational procedure it is also necessary to adjust the general scale (variance) 
of the trial values to agree with that of the observations. Generalizing these 
computations we have the following steps for an iterative computation of 
scale values: 

І. Select 7, , any set of trial S values. The values 0, 1, 2,3, 4, -++ , (n — 1) will suffice 


although convergence will probably be slightly more rapid if the average difference of paired 
D;;-values is used. 
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. Compute MT, . 
. Adjust the general scale of T,, to Z by computing 


со to 


a? = 72'7/(Т{.М'МТ,.). 


4. Set T, = aT, , and MT, =аМТ,.. 
- Compute the error of approximation 


e 


ME, = Z — МТ. 


6. Compute the correction 


€, = NU + LNME = NI +I — MN)ME,. 
т. T, = т +0 b, 


where b is an additive constant to adjust the origin to some convenient value. 
Repeat beginning with Step 1 until the error of approximation obtained in Step 5 is 


negligible. 

As a guide to computation it should be noted that C is computed most readily by the 
following sequence: multiplying to obtain NMZ; ; selective addition of elements in NME, 
yields LNME, ; another addition of ME, to LNME, gives (I + LN)ME, ; multiplying 
gives N(I + LN) ME, . In two problems worked by this procedure each element of the vector 
indicated in Step 5 was reduced to .005 or less on the third approximation. One problem is 
presented here to illustrate the results of this procedure. 


Illustrative Problem 


A food preference questionnaire was constructed using five different 
main courses and the ten composites formed by taking all possible pairs 
of these five. For each choice of the form (i vs. /) or (? and j vs. g) or (t and j 
vs. g and h) the subject was asked to indicate his preference. Three sample 
choices are shown in Table 1 together with the code used in Tables 2, 3, and 
4 for each of the five foods. А 

This questionnaire was given to 92 college students with the result 
shown in Table 2. The number in each cell indicates number of votes for the 
item indicated at the beginning of the row when it was paired with the item 

hen given the choice between Tongue 


at the top of the column. For example, w 
p hose Tongue. Four of the paired 


and Pork hose Pork while 24 ¢ 
rk, 68 persons chose gene These were utilized rather than 


Ine вапна were made by only c 
озегі f complete cases to дә. А 
ате А ; vs. 1) were omitted from the question- 


C 9 ji and А 
omparisons of the form (7 and 7 been interesting to have had 


naire. As the results turned out, it would have been т 1 
such items. A few comparisons of the form (# and j vs. ? and К) че как 
These, however, were eliminated in the analysis since it was not clear wo А 
results whether the subjects were judging one pem Pa = к 
Were “snoring the common element an comparing (ў vs. /) · 

Пресня. d S vs. T and L) was given 1n the middle 


(1 vs. 1) comparison. The item (P an 


* 
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TABLE 1 


Sample Questionnaire Itens 


Code used in 
Tables 2, 5, and § 
OE SI ARDEN, 


Roast Rib cf Prime Beef (в) 


Roast Loin of Pork (Р) 


Boiled Stoked Beef Tongue 
Loin Lamb Chop L) 


Roast Rib of Prine Beef В 
59 Sirloin Steak S 


Sirloin Steak (s) 
ho. 
Roast Loin of Pork Р 
Boiled Szoked Beef Tongue T 
TABLE 2 


Food Preference Data* 


* Number of votes for stimulus listed at side wh 
with the stimulus listed at the top. 


en it was paired 
The key for stimulus abbre- 
viations 18 given in Table 1. 


AMI V Ne 
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of the questionnaire and again near the end with the results shown in Table 
2. In converting this cell to a “distance” entry the average of the two different 
distances was used. 

The values shown in Table 2 were converted to normal deviate values 
by use of a table of the normal curve. An adjustment for different discriminal 
dispersions was made on the basis of a priori considerations. Since the number 
of items involved in the three types of judgments [the (1 vs. 1), the (2 vs. 1), 
and the (2 vs. 2)] were in the ratio of 2 to 3 to 4, the variances, or the squares : 
of the discriminal dispersions, might reasonably be assumed to have the 
same ratio. To approximate this ratio the entries from the normal table were 
used directly for all comparisons of the (2 vs. 1) and (1 vs. 2) type. For 
comparisons of the (2 vs. 2) type these values were multipled by 1.2 (which 
equals approximately V4/ 3). For comparisons of the (1 vs. 1) type the 
normal curve values were multiplied by 0.8 (which equals approximately 
V/2/ V3). 

Also, the three types of judgments were scaled separately and the 
Scales seemed to agree reasonably well after the adjustment for the assumed 
differences in discriminal dispersions were made. { 

Judgments of the form 92 vs. 0 were converted into "lower bound" 
values by assigning the normal deviate for 91.5 vs. 0.5. As the results turned 
Out, these values were not systematically lower than predicted values so 
perhaps such an approximation is possible where only a few 100 per cent 


Judgments are found. : 
Table 3 gives the resulting 
each missing entry. The sums 0 


the column vector Z. i i 
The results of the computations indicated in Steps 1to7 are shown in 
Table 4. Since a in Step 3 turned out to be nearly unity each time, only the 
trial vectors T, , T; , and T; are shown while T; , Tar, and Т, are not shown. 
3 


ors indicating imati least squares fit 
lhe v indicati he error of approximation to a les р 
т cate extremely rapid convergence. 


ME 7. are shown, and indi trer à 
uM Palace Lee C, + b, and C; + b; indicated md " ar as ` 
shown. It should be noted that each b; has been aur it ; 2 ү m: 
Corresponding to “Tongue and Pork” (TP) remains а € MS sen 
adjustment facilitates comparisons of the successive approxi 
not affect goodness of fit since the origin is arbitrary. acm 
A very interesting regularity appears In these es m ete 
in order from least preferred to most preferred are Е ТК 2 d 
Beef, and Steak. Adding Lamb, Beef, or Steak д л а 
posite; while adding Tongue or Pork decreases the уа xn SANI. 
Thus, the evidence, ‘purely from the general ordering n А 
Tongue and Pork as negative and Lamb, Beef, and Stea + e nee 
A more precise determination of the zero point will be giv 


D matrix, provided a zero is substituted for 
f the rows shown in Table 3 are elements of 


TABLE 5 


Food Preferences 


Experimental Inter-Stimulus Distances 


(The key for stimulus abbreviations is given in Table 1) 


TS 
TP -0.850 x x -2.020 X -1.920 -2.020 -2.052 -3.060 
m -0.512 -0.940 -0.856 x -1.360 -1.832 -1.290 -1.840 -2.040 -2.290 -2.290 |-15.250 
TL x x -1.270 -1.600 -1.380 X -2.020 x -2.208 | -8.78 
P -0.592 -0.450 x -1.368 X -1.840 -2.040 -1.510 -1.600 | -8.838 
тв -0.160 x X x -1,080 X -0.940 -1.548 x -3.838 
PL 0.940 x x 0.300 о x -0.036 X -0.710 x x -1.120 x -2.h2h | -3.050 
L [0.850 0.856 x 0.592 0.160 x Ba ою -0.410 -0.856 -0.710 X -1.208 х -1.230 | -1.956 
T| x x x 0.450 x 0.036 0.000 Mio and -0.390 X -0.768 x x x -1.176 
PB] X 1.360 1.270 X x x 0.410 0.504 о х x X -0.Т00 -1.176 x 1.628 
B [2.020 1.832 1.600 1.368 x 0.710 0.856 0.390 go о -0.300 х -0.656 -0.670 x 1.150 
Р5 - x x 4.664 
LB|1.920 1.840 x 1.840 x x x 0.768 x x was Om -0.250 x x 6.214 
5 0.740 0.656 x nage о X x 13.034 
IS 1.176 0.670 x x г зш * 9.246 
BS х х X 12.812 
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TABLE В 
Iterative Procedure for Determining Scele Values 
fee | 


-.09550 


24 — .346 -.07185 
-.16190 
PStimuli abbreviations аге defined in Table 1. 


T T, а T, are succe (scale values of the stimuli) 
ant 
$5 ә 3 


ssive trial values for 5 


$ 

^ ME, аге the errors of approximation. 
MES ‚МЕ, and ME, 

Л and C, are the correction terns. 


t vi 
It turned out to be necessary to carry the computations 9, зеш xs five at 
decimals in order to be certain of having жаб мй A 
to three decimal places. Amount of lowes ate Т АЕ 
element in C so that correction on lowest 


flt but merely keeps origin of т, 


| 4 
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PROPORTIONAL PROFILES, AND LATENT STRUCTURE 


W. A. GIBSON 


PERSONNEL RESEARCH BRANCH 
DEPARTMENT OF THE АВМҮ* 


The identity of problem and solution in Lazarsfeld's latent structure 
files is pointed out. Anderson's latent 


analysis and Cattell's roportional pro 
structure solution is ac apted to proportional profiles to yield a possible solu- 
tion for the communality and rotational problems in factor analysis. А numer- 


ical example of the latter is provided. 


A Formal Identity 


Two recent articles (4, 6) have presented identical solutions to the same 
geometric problem which has appeared in two different contexts. The two 
different contexts are Lazarsfeld’s new latent structure model (7) and 
Cattell's concept of proportional profiles in factor analysis (3). The common 
problem may be stated as follows: 


Given two Gramian matric 
m, to find corresponding ort 
V; , that are proportional by col 
from theoretical considerations. 
Let F, be an arbitrary orthogonal factorization of Ri , and let A be the 
orthogonal transformation from F, to V; . Also let D be the diagonal matrix 


es, R, and Rz, of order n and rank 
hogonal factor matrices, V; and 
umns and are known to exist 


of proportionality constants. Then ea 
В, = Е.Е: = Vii» (1) 
Үү, = Р.А, (2) 
V: = yiD = F,AD, (3) 
and 
Ry = ViVi = ViDVi = FAD AT . (4) 
The solution is to form the matrix : 
-1 
А = (FUR, FRF PP) 
= (FIP "FIF ADA FEED (5) 
= SAU. 
P. orth Carolina and completed at the 


EA iversity of N 
was initiated at the Univer доев; 
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factor it by any principal components method, and normalize the resulting 
factor matrix by columns to obtain A. Then V, and V, are given by (2) and 
(3) in turn, D having been determined in the normalizing process. 

Since the diagonal elements in D^ are the characteristic roots of А, 
this solution cannot yield a unique A unless all diagonal elements in D? 
(hence also in D) are different. This creates no serious difficulties in latent 
structure analysis, for there the empirical data can usually be combined 
in such a way as to make all diagonal values in D? quite distinct from each 
other. In proportional profiles, however, there is no such manipulatory 
freedom. There R, and R, are correlation matrices for the same tests based 
on two different samples, and V, and V, are corresponding factor matrices. 
The diagonal values in D indicate differential degrees of selection on the 
factors in samples 1 and 2. In practice it may often be difficult to find two 
samples in which the degrees of selection on the various factors are all quite 
different. This will ereate problems of slow convergence or even indeterminacy 
for some of the factors. While such problems will not be discussed here, their 
possible seriousness should not be minimized in evaluating the notion of 
proportional profiles. 

It is instructive to consider the special case where F, is the principal 
components analysis of R, . The matrix FiF, then is diagonal and contains 
the characteristic roots of R, in its diagonal cells. The matrix (FiF) is 
also diagonal, with the reciprocals of the characteristic roots of R, as its 
diagonal entries. The pre- and post-multiplication of the matrix F(R,F, by 
this diagonal matrix in (5) produces a situation which is exactly the opposite 
of what might be expected. The last diagonal values in (ГІР!) are the 
largest, so that the last vectors in A are likely to be the longest. Hence the 
first principal components in A are determined largely by the last principal 
components in R, . Since the last components in R, are probably most 
seriously affected by error, less confidence can be placed in the first columns 
3 V, and Va than in Ше last. This line of reasoning applies also to the centroid 

" insofar as it approximates the principal components E. 


Anderson's Latent Structure Solution and Proportional Profiles 
Since the geometric problem in 1 
portional profiles is the same, it follows 


solution for the former, developed by An 
(5), has the advantage over Green's sol 
any unknown elements in the manifest 
terms). It is not difficult to adapt this 
to eliminate both the communality and rotational 
if Cattell's concept is accepted and is workable. 
Let the two correlation matrices, R, and R; , be rearranged, if need be, 
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so that the two sub-matrices of order n — m by т in the lower left corner of 
R, and R, have rank m. Let these two sub-matrices be designated P, and P,, 
respectively. Let the subscript a refer to the last n — m test variables, and 
let b stand for the first m test variables. Then 


P, = VaVi, (6) 


and 

Р, = Vaz Viz = YaaD Yh . (7) 
That is, У and Уш are made up of the last n — m rows of V, and V; , re- 
spectively, and Vi, and Vi, are square matrices consisting of the first m 


columns of У; and У: , respectively. 
Now form the matrix 


B = (PIP) "PIP, 
= (Va Va Va V) Von Va Vai D Via (8) 
= VRV a Va) VuVaVaVaD'Va 
= Vir D Vi . 
The next task is to obtain the characteristic roots and a set of right-sided 


characteristic vectors of B, for it turns out that the roots are the diagonal 

entries in D? and the vectors are the columns of V 4; K, K being an arbitrary 

unknown diagonal. Thus 
BVK = Vi’ D Vha Vi К (9) 


= ҮРК = Vir KD’. 


1 


Once a matrix V; 'K is obtained, (6) can be post-multiplied by it to give 
(10) 


C = РК = VaVh К = VaK. 
Thus V,, becomes available except for unknown multipliers on its 
columns. It happens that these multipliers are quite readily obtained in 
latent structure analysis because of the way in which the ешр Ed 
matrices are bordered. It is necessary to adopt a different md p i 
portional profiles. Let the symmetric sub-matrix of order n — m in the 
right corner of R, be designated Q, . Then 
Q, = Vala = (Va KKK Va) = СКС". 
іх Q, is entirely given, as 15 the matrix 
diagonal terms in Q, are given, and 


(11) 


Except for its diagonal terms, the matr: 
C. Assume for the moment that the 


define P 
G = @(С'0) (12) 
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Then К” can be obtained from (11) as follows: 
K” = (C'C)?C'Q.C(C'C)" = QG. - (3) 


Given K^, К^! is easily formed, after which V, сап be determined by 
rewriting (10) in the form 


Va = CK", (14) 


This suggests a simple iterative procedure in which V a1 is first approximated 
by inserting rough diagonal estimates into Q, and applying (13) and (14), 
then improved diagonal estimates obtained from the first V,, permit a second 
cycle of the same kind, and so on until further iteration makes no important 
change in V,, from one cycle to the next. For large Q, matrices, little or no 
iterating will be needed, since the products involving the estimated diagonals 
will constitute such a small part of the sums of products. With small Qı 
matrices the iterations will not be very time-consuming. To save time in 
iterating it may be worth while to express Q, as the sum of two matrices, 
one of them being a diagonal matrix containing the unknown diagonal 
elements of Q, , and the other being О, with zero diagonal elements. Designate 
these two matrices E and Q,, , respectively. Then 


К = 600, + EG = G'Q4G + G'EG. (15) 


Only the last term in (15) changes from one trial to the next, and it is rapidly 


formed. The matrix G remains constant throughout the iterative process. 


Given Va , Vy, can be solved for from (6), (10), and (12) as follows: 
Va = PIVA(V4 Vua) 
= PIVaK(KVAVAK)?K (16) 
= PIC(C'C)"K = рК. 
The last member of (16) has been 
G that is used in the iteration to 
two purposes. 

Once V,, and V, are known, all that remains to be done is to form Vi 
from them to give the “true” factorization of R, , and then to compute Vs 
from the first part of (3), the matrix D having been formed from the square 
roots of the characteristic roots of B. The goodness of fit of the two factor- 
izations V, and V, is indicated by the agreement between the first and third 
members of (1) and between the first and second members of (4). 

It is to be noted that this adaptation of Anderson’s solution to the 
problem of proportional profiles assumes the number of factors to be known 


at the start. This is not a serious drawback, however, for there are many ways 
of estimating the rank of a correlation matrix, and the inverting of P(P, can 


adjusted so as to involve the same matrix 
determine V,, . Thus G is made to serve 
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be done in such a way (cf. 8, pp. 46-48) that a change in its order can be 
accomplished without serious loss of computing time. 

Several equations in this paper have least squares properties that should 
perhaps be listed explicitly. The first is (5), which is the best fitting solution 
for AD*A’ in (4). The next is (8), which is the least squares solution of В in 
the equation P, — P,B. Then (13) is the best fitting solution for K^ in 
(11). Finally, (16) is the least squares solution for V, in (6). 


A Fictitious Example 


As an example of the present solution for proportional profiles, consider 
the two fictitious correlation matrices, R, and R; , that are shown in Tables 
land 2. They were generated from a simple structure V, and V, that originally 
were strictly proportional by columns, but that subsequently were “blurred” 
by adding small random increments to each of their entries. Thus there exists 
no perfect proportional profiles fit for R, and №, , so that the various least 
squares properties of the solution will have definite advantages in this 
example. i 

Inspection of R, and R, suggests that three factors will probably account 
for both, and that the 7 X 3 sub-matrix in the lower left corner of each 
probably has rank 3. These two sub-matrices are therefore designated Р, 
and P, , respectively, and (8) is used to form from them the matrix B that is 
shown in Table 3. 

Table 4 is the matrix D", containing in its diagonal cells the characteristic 
roots of B. These characteristic roots are obtained by forming and solving 
the characteristic equation of B(cf. 8, pp. 96-28 and 44-45). The algebraic 
Sign of the characteristic roots in Thurstone's discussion is the opposite of 
the convention being used here. The three columns of Table 5 are a set of 


right-sided characteristic vectors of B. Column A, for example, is а solution 
fficients matrix is the 


to the set of homogeneous linear equations whose coe х 
matrix В with its first characteristic root subtracted from each of its diagonal 
cells (cf. 2, pp. 250-251). Column B is the same thing using the second 
Characteristic root, etc. Each such set of homogeneous linear equations 18 


readily converted into a consistent set of m non-homogeneous linear equations 
of the unknowns at some соп- 


in m — 1 unknowns by arbitrarily fixing one ns 
venient value. Here the first element of the first characteristic vector was 
Set equal to unity. The resulting over-determined system can be EA by 
any method. Here it was done by least squares because of the slight in- 
consistency resulting from rounding. ^ 

The ОК step is to form the matrix C from Р, and Үн K by а of 
(10). C is shown in Table 6; (10) indicates that it 15 proportional by columns 
to Va. 
The task now becomes one of extra 
K=, As a first step the matrix б is forme 


from C by determining 


ting Ve à 
Pes G is shown in 


d by means of (12); 


Охо ON OU FWN I 


m 


TABLE 2 
A Fictitious R, 
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TABLE 1 
A Fictitious R 
c MM————Ó—À————— ЕБЕ —— — —P——— 


1 .08 .06 .32 ..23  .13 .3T 
2 +590 .01 5 „10  .05 „ЗТ 
3 -10 .20 .07 ah i90  .15 
} .10 .05 .37 .26 щш HD 
5 -05 40 12 .2т  .3h 
6 .05 302 20 „16 „10 
T К s s40 „02 19 «62h HT 
8 : : .19. 420 39 .23 .28 
9 х ФТ .16 pi әз .28 
10 Ж „42 7.35.10. ат [28 28 
TABLE TABLE } TABLE 5 
lam - е leo 
B=(P/P,) PrP, D Voi 2K 
See ©“ а 
= 1 2 3 n B С А 6 
І 1.76 29  .85 А 1.80 I 1.00 -.33 -eL 
2 07 i.15  .00 B 113 2 21: 1.00 02 
3 10% 118 755 c 54 3 x02 3.33 2,00 


C=P V} fg G=c(c/c)~1 
SF m 
a: ИБИ. ИН ——— FE 00 


Table 7. Now the constant part (G'Q,4G) of (15) ean be formed; it is shown 
in Table 8. 

` The initial diagonal estimates for Q, might as well be the same first 
approximations that have served so well with centroid factoring—the highest 
non-diagonal entry in each of the columns. These need not be taken from 
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y do for 


within Q, . They can come from the first three rows of №; , as the: 
pears in 


columns 4, 5, and 6. The resulting variable part (G'E,G) of (15) ap 
Table 9. The subscript on Е, indicates that it is the first in a series of Ё 
matrices. 

The sum of the matrices in Tables 8 and 9 is given 
vanishing side entries in Table 10 call attention to 


by Table 10. The non- 
the fact that there is 


TABLE 8 TABLE 9 TABE 10 , 
4 “c= 
6010 G/E,G K,~2=6 0,640 EG 
A (c А B C A B C 
A ga 01.18 83 A. 2.72 -1.17  --63 A 3.64 -.04 .20 
B 1,23 37 2} В:=1„17 2.08050. B sow eels ELT 
c 3 24 1.61 C -.63 -.0T. 1.1 Ci n20 17 2.77 


nothing in the equations which prevents this estimated K^ from being 
non-diagonal when the data are imperfect. Some reduction of the side entries 


in K^? may occur as the iterations proceed, but they may never vanish 


identically. 
A first estimate of K^' is obtained by taking the square roots of the 


diagonal entries in Table 10. Thus the side entries are not used. This suggests 
that a further shortening of the iterative procedure would be to compute 
only the diagonal terms in (15). 

The first estimate of Veı , obtained by (14), is shown in Table 11. The 
last column of Table 11 gives the new diagonal estimates (the sums of 
squared row entries in this first Vai) which make up the E; of the second 
trial. Now the matrices G'EG, Kz”, Kz 1 and CK;' can be formed in turn 
exactly as in the first trial, the resulting second estimate of Vai providing 
an E, , etc. In the present example this process was continued for a total of 


TABLE 11 TABLE 12 4 
First V431-0Ki-l Final Vai=CK4 
ee 


ee з ME ce 
кш E REP 
d .50 -.02 -.0 126 TES 
Git 213 КЫЛЫ Е Ао тийе Hae -24 pL 
GU! бтз со REDDE te CU бз ЛО aes 15 
П Еее © о -12 -38 
в оой E a N à 129 -:08 2) :38 
ОИ А ДО: "оз от 2805 "35 
ТОИ T2 INANCO 2181 OS 10.42 -29 „29°... 


tical with £, at the level of 
in the estimated Va coul 


four trials, at the e 
] diagonal 


accuracy being used, 8 1 
take place. This final Va is shown 1m Tablo e ЫН 

i i hat no essential с 
estimates for Q, . It will ап tes but the first have 


the first to the final Им, 
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changed. Most of the changes occurred between the first and second estimates 
of Va . All changes after the second Va (six loadings and two diagonals) 
were of size .01. This suggests that one repetition of the iterative procedure 
may often suffice for practical purposes. 

Now V,, is formed by means of (16) and is recorded in Table 13. Tables 
12 and 13 together make up V, , so that V, can now be obtained by (3). V; 
appears in Table 14. The structure of V 


ı and V, is further removed than was 
anticipated from the simple structure 


configuration which generated this 


TABLE 13 


TABLE 14 
Vpi =P {ck ц Үр=ү;р 
——— ——Є—=—————-— 
B С 
1 .50 -.07 -.03 + б. 
ЕШ Ege "AO ST =.07 -.02 


3X. .10' -,02 ' 759 
зо =302 „у 


XO ON OW +оо IO 
ы 
К] 
= 
o 
m 
со 


Further rotation of V. 
and V, w impr ir si 
appearance, but that would of em Pins Sy em TU» ылы. 


› then the required 

Something like the 

“up of the parti 18 at least partially 
ry. 


arti 
The degree of fit for the present a test batte 


“VO Correlation matrices 
Pirical data, on the other 


____ ы е 
ee eee 
— a, A 
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ha: T i 

ee oa Le я sampling and sample size may appropriately influence 

ы E 8 to be expected that R, will be fitted best, since P, and Q, are 

tava Ша gite e Gao £ Lie eee 

2 r than the rest >, Since Py i 

ie that is used in the solution. If it were voee pop jen 

: e, the fit of the two correlation matrices could be more nearly equalized 
y post-multiplying C by D to form УК. Then the iterative procedure 


TABLE 15 
WE =——_ 
5 = 6 8 10 


oe M -.02  .01 

-. E *00 100 -.01 .00 „00 .00 = 

оо 01 „00 зол Ol .00 01: 1 30 
62 501 s02- .01 „02 


-.01 -.01 -.01 .01 .02 
:00 100 100 .00 .01 .01 Е 
‘оо 100 00 .01 .02 -.01 ‚92 .02 01 
бо бо бо Ol „бї 01 Ol .02 102 

00  .01 01  .0? 


i TABLE 16 
a 
1 2 3 4 5 6 Т 8 9 10 
1 de so 01 .01 o 2.08 -.03 .00 .02 
2 .02 оз 103 -.01 -.02 0l ton .02  .01 
2 90 -.03 O2 от -.03 03 02 «09 2.02 
101 03 «02 фо 200 „Ol .02 .01 .9 
é OI 1.01 sêl 00 tog -.05  .0& .02 -.02 
от -lo2 -.03 .00 .02 -,03 -405. =.0% -=%03 
28001 101, 2:03 «0L зб 2:04 Тоб -.01 -.01 
-.03 .01 .02 .02 Soh -.05 -.02 J00 sod 
ue "о ы 00 01 05 -294 -.01  .00 .01 
=O — «Di. 09 


K^! based on the Q section 
tric mean) of the two X 
wo correlation matrices 
tional profiles 
t be factored and the 


btain a second estimate of 
(possibly the geome 
lution fitting the t 
he Cattell solution for propor 


could be applied to o 
of R, . Some sort of average 
estimates should then lead to a so 
to about the same extent. T 
probably provides better over- 


communality problem reappears. | 
It may be mentioned, in passing, that the 


ВДВ, applicable at nearly every stage of the presen 
omitted here to simplify the exposition. 


all fit, but then R, mus 


usual summational checks 
t solution. They have been 
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A NON-PARAMETRIC TEST OF CORRELATION 
USING RANK ORDERS WITHIN SUBGROUPS 


Warren S. TORGERSON 


EDUCATIONAL TESTING SERVICE 


; Kendall's rank order test for association between two variables is general- 
ied to the case where the total sample is made up of several subgroups and 

e data on one or both variables consist of the rank order within each sub- 
RUD. The test involves no assumption: of measurement, 
Е враз of distributions, or relative level of excellence or amount of variability 
of the different subgroups. Two empirical examples indicate that the normal 
approximation to the exact test of signi can be considered adequate for 

ideration is given to the case of tie 


most practical situations. Special const 
2 only a slight 
e ties in bot! variables within 


ranks. If ties occur in but one variable within any given subgroup, 


modification in procedure is needed. Exter ] A 
subgroups lead to difficulties in determining the appropriate correction for 


continuity. 


I. Problem 


ү The problem occasionally arises of determining whether or not à sig- 
nificant correlation exists between two variables when the total sample is 
made up of a number of subsamples, and the scores on one or both variables 
consist of rank orders within the subsamples. Such a situation frequently 
occurs, for example, when one wishes to determine whether or not scores on 
a psychological test are significantly related to supervisors’ rankings with 
respect to proficiency on some task. Similar situations commonly occur in 
the military establishment. In the measurement of personality traits it is 
also desired at times to determine the significance of correlation between & 
trait measured by а personality schedule or test and ratings or rankings on 
the trait as made by peers. In any of these situations it frequently happens 


that the population is best considered as composed of many subgroups, each 
headed by its own supervisor osed of subgroups whose 


or officer, or comp 

members are well acquainted with one another. 

Let us consider the first situation in somew 

a total sample is composed of k small subgroups 

Seven subjects each), еас 

place his own subordinates i Р 

tion concerning 

information is 


task of interest. No supervisor, E 
proficienci pjects su ervised by others. Hence, nO i 
iom TAE E ts from different subgroups. 


available concerning the relat: erson in One 
It is not known, for examp 3 


hat more detail. Assume that 
(of from, вау, three to six or 
Each supervisor can 
proficiency 01 the 


ive positions of subjec 
le, whether the highest ranked 
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subgroup is better or worse on the criterion than the lowest ranked person 
in another subgroup. 


When one is confronted with this problem, perhaps his usual procedure is 
either to decide not to carry out the experiment after all or to use procedures 
which require one or another of several sets of simplifying assumptions. 

One such procedure, for example, would be to throw out enough obser- 
vations so that each subgroup contains the same number of cases, treat the 
ranks as if they were а single rank order with Ё ties at each rank, and then 


IL The Test of Significance 

The proposed test is ап extension of Kendall’ 

two rank orders (2). His test involves, first, com 
and, second, the determination of the probabi 


Usin 

g the norm: 

о " al approxi i 

f the critical ratio аак 
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possible valu 
between es of the sum under th 
the two vari P r the null hypothesi 4 
n(n — 1)(2n + Merge is symmetrical about zero EA iieri 
rapidly, 25 dien та a mie the distribution approaches ч m» 
equate results. Ex г more subjects, the normal a) PER y very 
Q, p. 141). "Exact distributions have been ener Pam Шоц nm 
The Kend ae 
А all sum, of А 
а single БП, of course deals with r k 
group. W Д ank orders of two v i 
Here we find oH нта : wiped concerning hh pre 
Lets, de S ndall sum has some exceeding! leasi PM 
; denote the Kendall sum of the jth subgroup, hers E 
yD, oe e 


We сап 
obtai d 
SENE in à sum of Kendall sums for the К subgroups Let S denot thi 
. e this 


У (1) 


The distri 
stribution of S 
eadh saber І under the null hypothesis of zero correlati ithi 
E : on 
dier itid 5 д 18 also symmetrical about zero and has а paces ess 
variances of the sums (23) of the separate subgroups, 
3/18. e 


Furthe = 
т ЛУ К 
‚ the distribution also appears to approach normality very rapidly. 
ignificance 


the test is simply ® 


The tono? whieh? 
," which is subtracted from S when 
for continuity- Since pos 


S Ww 

whenev ; 

of S i er Sis negative, is а COT 

With n е given situation аг i 
ass interval of two, the correction 


th : 
e js interval, is unity- 
о H 
get an idea of whether or not the distribution approaches normality 
3, two empirica were run—one using 
ps of sizes 4, ^ 


rapi 

ily enough to о аш б 

5, and » p f sizes 3, 4, 45 and 6, and the other using subgrou es 

value of S or every possible value of S the exact pro abilit of obtaining 8 

probabilit that large or larger was cO uted, along with the corresponding 

Table 1 ji using the normal approxi а 2 ће results 

аз Е s can be seen, the approxim: 1 3 

subsam e these. Since most practical jtuations V 
p eno subsamples of greater size, the normal ар 


considered adequate. 


proximation 
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TABLE 1 


Comparison of Probabilities Computed Using the Normal 


Approximation (Py) with the Correspon: 


ding Exact Values (Pg) 


ПІ. Numerical Example 


We shall apply the t 
The total sample is made 


; : L2,...,5 f 3, 5, 6, 8, 
and 8 subjects respectively. For each subject in e Aur i 


é т ach subgroup, two scores 
› the score on variable 1 being expressed in terms of the rank 
position of the subject wi 


nd the score on variable 2 
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TABLE 2 


Raw Data for Numerical Example* 


Group 1 Group 2 Group 3 Group h Group 5 
Wi Va X: Y) w Vo n h Яз a 
i Б 1 h6 1 б 1 6 i 6 
2 62 2 9h 2 17 2 38 2 То 
3 56 3 65 3 51 3 шт з 68 
h Lo h 17 у щш h 59 

5 32 5 56 5 3 5 18 

6 19 6 37 6 67 

7 59 a БЇ 

8 30 8 Sh 


*Scores on variable 1 (Vi) are expressed in terms of 


rank order within subgroup and on variable 2 (Vg) in terms of 
test scores, 


expressed in terms of ordinary test scores. The subjects have been arranged 
In order of their rank position within their own subgroup. 

‚ The first step is to transform the test scores into ranks within subgroups. 
This has been done in Table 3. We now compute the Kendall sum for each 
Subgroup. Since the subjects in each subgroup have been ordered with respect ' 
to variable 1, this is accomplished most easily by first determining for each 
Subject in the subgroup the number of subjects below him who have a higher 
tank number on variable 2. For example, in subgroup 2 there are two subjects 
with higher rank numbers below the first subject, two below the second 
Subject, two below the third subject, one below the fourth, and none below 
the fifth. If the sum of these numbers is denoted by т, then 


s = 2r — in(n — 1). 
For subgroup 2, 
r=2+2+4+24+1=7, 
and, hence, 
s = 14 — 3(5)(4) = 4 


In like manner, s, = 3, 8: = 7, s, = 8, and s, = 12, and the sum of Kendall 
sums, S = 3 +4+74+84 12 34. 


\ 
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TABIE 3 


Computation of Values of sj and 5% 


Group 1 Group 2 Group 3 Group Ц Group 5 
Vi Va Vi Va Vi V2 Vi Va Vi Үг 
Lh gy 3 i 2 d. d 1 L4 

2 2 2 2 2 1 2 6 2 i 

3 3 3 1 3 n 3 5 3 2 

ho L h 6 h 3 n 5 

B Б = 3 5 L 5 8 

6 5 6 1 6 3 

7 2 7 7 

8 8 8 6 


rT) = 2+1 = 3 8) = 6- 35 3 
To = 2424241 = 7 $5521, -10s | 
ту= Цяц+2+0+1 = 11 835 22-15» 7 
Ys 7+2+2+3+2+1+1 = 18 арт 36 - 28 = 8 


Tc 1465523504240 = 20 Sp ho - 28 = 12 


EET 
*Soores on both variables 


expressed in terms of rank 
order within subgroup. 


The variance of S under the null hypothesis is equal to the sum of the 
variances of the five separate subgroups. From equation 2 we have 


os = Y&(06 + 300 + 510 + 1176 + 1176) — 179.3333. 
From (3), 


34 — 1 
CR = Ss = 2.4 
V179.3333 E 


which, using a two-tailed test, is significant beyond the .02 level. 


А 


3 


С 
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IV. Tied Ranks 


Heretofore we have assumed implicitly that ties would not occur. When 
ties do occur several alternative procedures are available. One such procedure 
would be to assign ranks within ties randomly and then proceed as if no 
ties had occurred. A second procedure might be to assign ranks within ties 
in the most favorable way and also in the least favorable way and then run 
a separate test of significance on each. If both tests lead to the same con- 
clusion all is well. However, this procedure is inadequate if the two tests lead 
to conflicting decisions. 

A third procedure is to assign the mean rank to the ties. This is the 
procedure adopted by Kendall for the single group case. 

The mean rank method, or mid-rank method as it has been called, 
requires some additional considerations in computation of both the Kendall 
Бш. and its variance for the individual subgroup. Two cases need to be 
distinguished: (i) that in which ties occur, for any given subgroup, in only 
one of the two variables; (i?) that in which ties occur in both variables within 
a subgroup. 


(i) Ties in one variable only for any given subgroup 


It should be noted at the onset that it is irrelevant whether the variable 
containing the ties is the same variable in all subgroups, or whether for some 
Subgroups it is one variable and for others, the other variable. 

. The Kendall sum, s; for any subgroup j has the same meaning as before 
with the additional stipulation that tied pairs, being neither in the same nor 
in the reverse order on the second variable, are counted as zero. If the subjects 
аге arranged in order on the variable containing no ties, the sum is the number 
of pairs on the second variable that are in the same order minus the number 
of pairs in the reverse order. Tied pairs neither add to nor subtract from 
the sum, 

Caleulation of the variance of the jth subgroup requires the following 
Modification. There may be several different ties, and any number of subjects 
may be involved in any given tie. Let a be an index referring to а particular 
tie (a = 1,2, ... , g) and £,; refer to the number of subjects involved in the 
ath tie of the jth subgroup. Then Kendall has shown that 


g 
9; = x С = 1(п; + 5) — >; ballar = 1) (2ta; + э]. (4) 
The sum of Kendall sums and its variance under the null hypothesis remain 
the sum of the individual sums and the sum of the individual variances, 
Tespectively. The distribution remains symmetrical about zero and Ei. 
appears to approach normality rapidly. Possible values of 5 for any particu. ч 
Combination of number and size of subgroups remain either all odd de 
even, with a step interval of two. Hence for the normal approximation, tne 
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correction for continuity remains one, and the critical ratio, 
Ср. лет зы. (5) 


where the individual subgroup sums and variances are computed as noted 
above. 


(ii) Ties in both variables for some or all subgroups 


The computation of the separate Kendall sums is the same as before 
except that ties in either rank do not contribute to the sum (see 2, p. 26, for 
more detail). The formula for the Separate variances is given below, where 
tai refers to one variable and иь; to the other. 


о = Ys[nimn; — 1)(2%; + 5) 
D lailla; — 1)(2, + 5) — 5 tilus; = 1)(2иь; + 5)] 


1 
ег Ini; — Тул, — 2) [> tei(ta; — 1)(4; — 2)] (6) 
TX wins; — D; — 2)] 


1 
+ Bat, ту 6s — 0157 уб, — 1)]. 


convert the problem to the 


t th "ties in one variable only 
case by assigning a rank order r: 


for any sub " р 
ROAPUOBIOUD andomly to the ties in one 


variable. 
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MAXIMIZING TEST VALIDITY BY ITEM SELECTION 
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„The exact condition for discarding k items from a test in order to obtain 
a residual test with higher validity is derived. A proof that validity always 
increases is given for the case К = 1. The lack of uniqueness of maximum valid- 


ity when ac ieved by use of the condition is discussed. With the use of addi- 
tional restrictions on items to be included in the initial test, a practical test 
construction procedure which has several advantages over previous methods is 
‘developed. The homogeneity of tests constructed by the method is discussed, 
and applications are given. 


The problem of selecting from among a large number of test items 
those yielding a test score which will correlate maximally with an external 
variable has been studied by a number of writers. Gleser and DuBois (6) 
and Gulliksen (8) have summarized pertinent research. The problem is of 
first importance when it is necessary to construct a test which has a high 
correlation with some known criterion; the same problem would arise, how- 
ever, if it were necessary to select those parts of a composite experimental 
criterion which would correlate highest with a fixed test. Mathematical 
solutions to either problem, or to the simultaneous solution of both problems 
(10), are known. These solutions are impractical in the case of item selection, 
however, because of the large number of partial regression weights required 
for items. ] 

An interesting method of testing composites for sampling stability is 
known (3) but its application for comparing numerous item composites 
would present serious difficulties. Recently Lord (12) has derived sampling 
variances for test statistics under conditions of item sampling, but presumably 
the behavior of the validity coefficient when there is sampling of both persons 
and items is still unknown. Guilford (7) advises that the approximate solutions 
be applied only with large tests which have been administered to large 
numbers of subjects. Richardson and Adkins (3) suggest that possibly апу 
item selection index would be so susceptible to sampling fluctuations that а 
choice among methods of selecting items would be practically а matter of 
no importance. Tukey (15) includes item selection for validity among the 


important unsolved problems of experimental statistics. а 
Because of difficulties inherent in exact solutions, and despite lack of 


precise knowledge of sampling fluctuations of item composites, investigators 
153 
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have devised various approximation methods. Toops (14) advocated the con- 
struction of tests starting with one highest correlating item and adding, one 
at a time, items which most increased the validity of the composite. Adkins 
has applied this method and she and Richardson (13) found that a much 


e of j samples what is gained by such 
reexamination may indeed not be worth the additional labor; but if the 


enough, it would seem worth while to 


| Horst (9) advises starting with a relatively large test and rejecting those 
items for which the ratio of a function of item-criterion covariance to a 
function of item-test covariance is small. The items are plotted and then 
selected using the functions as coordinates. Horst cautions against discarding 
too many items at once without recalculating item-test parameters, Gulliksen 
(8) has simplified Horst’s procedure somewhat, but it would still Been better 


to avoid plotting ifa more econo 
mical 
method of identifying items to be 


required. 


In the present paper the exact item i TUNES : 
the approximations Which are used can iesu рна is paene С 
same time to require less computation in applications ean i" at: ne 
ones. Also, a study of the selection condition itself reveals wh rie | ina al 
expedient of imposing a validity condition on individual it; bw bans ius d 
into a test is reasonable. ems to be admitte 
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I. Derivation and Properties of the Exact Condition for Discarding k Items in 
Order to Obtain a Residual Test of Higher Validity 


In order to construct a test with high validity, first a large number n of 
potential test items is examined for correlation with the external criterion 
С. It is then possible, with no loss of generality, to score each item j so that 
its covariance C,c with the criterion C is positive or zero. The sum of these 
n item scores, z, + -++ +2; +--+: + Ta , forms an experimental test T from 
which it is desired to discard items such that the residual test M correlates 
maximally with C. The correlation of T with C may be written 


tie = > Cic/SrSc = Cre/SrSc , (1) 


Where C's are covariances, S's standard deviations, and the summation is 
Ew the n item scores 1, --- , i, =+- , n. If the first k items, designated 
ој, +++, k, are discarded from Т, the validity of the residual test M is 


k 
тмс = (Cre — p» С,0)/8 8с . (2) 
Since M is to be more valid than Т, the condition 


Tre < Tuc (3) 
must be satisfied. Using (1) and (2), (3) may be written 


k 
Стє/8т8е < (Със — 22 Cio / 8x8c - (4) 
Making the substitution Sp_, = Sar , and multiplying by Sc , (4) becomes 


Cre/Sr < (Cre — x С;с)/8т-ь. (5) 


As à consequence of having scored items so that C;c > 0, the quantity in 
parentheses in (5) must, like the other terms, be positive. The terms of (5) 


сап be rearranged as follows, 


Sr-./Sr = К>: C;c)/C vc]; 


(X: с,д/бте < 1 — [Sr cd, 


k 
S,/Csc < (Sr — Sr-)/ 22 Cie - (6) 
Writing (6) in а form which will be more useful later, 
k 
Стс/8т > (>) C;o)/ (v — 87-0). ig 


untered in using (7) as & condition to be 


The practical difficulty enco TJ Bs. a c 
satisfied by rejecting k items in order to enhance validity lies in the fact that 
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the КА of S;., contains the sum of inter-item covariances of d 
items. Use of inter-item relationships is laborious and seldom feasible in t e 
typical test construction situation where there is available at most a machine 
for counting item responses. Further study of (7) is also warranted — 
it is not clear whether any method exists for identifying uniquely a set о 
k items to be rejected. , Ө | к. 

Squaring (5), substituting Уз, = V, + V, — 2€,, , and simplifying 
gives 


right in (6) are positive, the I: items (including the case Ё = 1) may not | 
dity unless S; > S7_,. Bu 
a negative quantity when S; > 
rearranging terms, we obtain 


x — 2C, in (8) is equal to Vra — Vr y 
8;-, . Multiplying (8) by — 1 and 


k k 

Ore Se О н Ойле — Y бугу (9) 

RANG i i ott 5. 

URN TS ET > Ty 

Although (9) is the general condition 

from which it follows that 2€,, > V 
to obtain a more valid test, the case 
a single item j, (9) reduces to 


(under the restriction, C;c > 0, 
1) for rejecting k items from T in order 
where k = 1 is more useful. When Ё is 


tuting $,8, , = (7. + Vr_,)/2; 


ccessively, increases validity. It 
st inequalities in the recursion 


series for (7), when single items 1,2, -.. are rejected, are 


If the terms on the left of the inequality signs in se 
by the constant Sc , they would become th 
coefficients as the test becomes shorter. 


quence (11) were divided 
€ correlations which are the validity 
These terms therefore comprise & 
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sequ 1 1 
^w. ence with а maximum upper bound equal to Sc . Every bounded mono- 
ne sequence is convergent, and 


Cre/Sr , Cor-ne/Sr-1 , p (12) 


isb 
: "Pie and can be shown to be monotone. Given that the first inequality 
) holds for item 1, we first prove that 


Cre/Sr € Ст-ус/ т-а + (13) 


By substituti 
in - = i ; 
also he inh Cir- = Cre — Cic and clearing of fractions, (13) can 


SraCre € SrCre — SrCic . (14) 


But this i pm 
n another way of writing the first term of (11), and this proves that 
pci м va 1 has increased validity from rre to rcz-oc . By induction, 
ван Get si rejected by (11), the validity always increases. The 
s therefore m n i ў 
(12) the fos onotone and convergent. By comparison with 


Cic Cic 

Beds" ae 7 om 
Which i 
m bows by the terms on the right of the inequalities in sequence 
Written MR ei but not necessarily monotone. Conditions can be 
ay мрн will make (15) monotone, but the writer was unable to find 
validity. were practical and at the same time would insure unique maximum 
N Р 

Р. агра (11) converges as it stands. (A difference between two 
ii am gent sequences also converges). This is sufficient reason to make use 
of pli sequence, one comprised of terms which are approximations 

), аз an aid in constructing valid tests. 


II. Development of a Practical Test Construction Procedure 

items from a test in order to maximize 
Je is in general not possible, and since 
hen both items and persons are 
ection restrictions in addition to 


бї Since rejection of a unique set of 
the ained) validity for a given samp 
н sampling variance of the validity w 
ampled is unknown, the need for item sel 
(10) is obvious. 
ite A practical restriction traditionally 
К should each have enough variance 
h mination of subjects. А second restriction, n the 
have individual validity, follows from (6). Reference to (6) when Ё is a single 
item j shows that because items are scored so that C;c > 0, discarding items 
for which 20% — V; = Vs = Vig <0 will not increase test validity. 
ecause of the scoring convention, therefore, the sign of Sr — ‘Sr-; is de- 


used in test construction is that 
to share appreciably in the dis- 
that items retained in the test 
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pendent upon the sign of C;c , which in turn is subject to sampling E 
tions. In order to be reasonably confident that these signs, and л. y 
the test scoring, will not vary in subsequent samples, it is necessary t и 
most of the items in the original test be valid. This condition alone could t 
satisfied by requiring that each item-eriterion correlation significan y 
exceed zero, or that 


Cie = S;Sez/ VN, (16) 


where 2 is a chosen critical value for a normal deviate, say 1.96. . 
Setting S; — 0.5, the maximum value, in (16) will provide a conservative 

test for all items for which S; « 0.5. That is, items satisfying (16) when 

S; — 0.5 are significantly correlated with C at the level specified by z, but if 


sated for by a higher covariance with the criterion. Ineluding in the initial 
test Т only items satisfying (16) when S, — 0.5, therefore, has the desired 
effect of insuring both that the initial test will 
most of them will have large variances. Reject 


one at a time is not feasible because 
the C;7 . A workable procedure, whi 
by Gleser and DuBois, is first to rejec 
Tecomputing C;;) and then to reex: 


are no further increases in validity. 


In order to secure a suitable approximation of (10), first note that 
Cic is the final term in the expanded numerator on the right, and that Cre s 
V; , and (% are in descending orders of magnitude. Substituting for the 


final C;c its mean value in T, Cic = Cre /n, and substituting the maximum 
value for V, , 0.25, reduces (10) to 


L2n / ) бе Cic 17) 
= Б, АК © Cir — 0.125 › ( 


of the value for V; in (17) is increasingly unimportan 

Suppose that m items for which (17) holds have 
a residual test M. Since (17) is progressively less accurate as items beyond 
the first are discarded, it is likely that some of the discarded items should be 
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put back into the test. Any rejected item j which satisfies the validity con- 
dition 

Tuc < Tanne (18) 
will be added to test M. By a development similar to that used in deriving 
(10), (18) may be written 


Chre > Cic(2C uo + Cic) 
Е с(2С› ic). i 
Vu 8 2Cix + V; (19) 
Using the approximations for Ce and V; described above, (19) becomes 
2m Cara: Cue 
las + i) V. $0, + 0.125’ (20) 


where m is the number of items in M. 

Items for which (20) holds are added to M to form a third test. It is then 
Possible to treat this third test as if it were T, the initial test, and again 
apply conditions (17) and (20); this process may be continued until no further 
Increase in validity occurs. 

It is also desirable to simplify computation of the covariances in (17) 
and (20). This can be done as follows: Test and criterion distributions are 
first transformed to comprise only five symmetrical categories containing 
the 8 per cent highest, 18 per cent next highest, 48 per cent middle, 18 per 
cent low, and 8 per cent lowest scores. A division of distributions into cate- 
gories containing 9, 20, 42, 20, and 9 per cents of cases is recommended by 
Flanagan (5) as maximally efficient when scores in the categories are assigned 
values 2, 1, 0, — 1, and — 2, respectively. The distribution used in the 
present paper will be slightly less efficient than the one recommended by 
Flanagan, but will have the advantage of possessing a unit variance. Using 
only five categories, the covariances in conditions (17) and (20) may be 
Obtained from item counts for the four extreme categories. 

To transform (17) and (20) so that the covariances are of the form 


Cli = (+ 9 2/0 = D/N, (21) 


where е, f, g, and ^ are frequencies in the categories in the order given above 


^ à z È d) 
(scores with zero weights, which are m the center category, are ignored), 
first note that the Menos of every such forced distribution Ho constant; 
V' = 1.00. Then if transformed values are indicated by primes, Sc = 81 = 1, 
апа 

rro = Ore/(SrSe) = „0108280 = Cre - (22) 


Only one transformed distribution corresponding either to rio ОГ to тут is 


needed; thus it is assumed that 


тут = С:2/08;8т) = 0!./08:81) = Chr/S; › (23) 
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and a similar expression can be written for е. Transforming (17) gives 


2n j Cic 24) 
(= = 1) > Gy (0125/85) ' ( 


and transforming (20), 


2m у Che З 25 
Б) аа 5 


A more convenient form of (24) is obtaine 
тє and, from (21), D/N = C’ to get 


( 2n Ere р, 


d by substituting У) 7"("/N = 


> ig. (26) 
2n — ] N Dir — (0.125N/S;) с 
where n із the number of items in Т. Similarly (25) is conveniently written 
" 
( 2m ) > Mc’ < Dio (27) 
2m +1 N Diac (0.125N/S,,) ’ 


where m is the number of items 


in M. Very little 1 
(27), especially when m/n > 8 


055 of accuracy occurs in 


and m > 30, if the value of Sy is taken to 
be т8т/п. 
The procedure for applying (16), (26), and (27) is outlined in the next 
section. 


ПІ. The Procedure for Applying the Item Selection Conditions 
1. Separate the answer sheets into five piles according to their criterion 
scores C, containing the 8 per cent highest, 18 per cent; next highest, 48 per 
cent middle, 18 per cent low, 8 per cent lowest C Scores, respectively. Mark 
papers in the four extreme C-score categories e, 59, 
they can be identified in Step 6. 
2. Record Оп item analysis sheets the values for e, f, g, and h, which are 
the frequencies of a response (for example, "true") in the highest, next 
highest, low, and lo 


west C-categories. Obtain from the item counts for the 
four extreme categories the difference 


апа A, respectively, so 


Die = 2 + f — (g + 2h) = NOI, 
for each item j [see (21)]. Next choose the direction of response for each item 
so that 2e + f > g + 2h that is, so that D,, > 0, and mark the items 
accordingly. 
3. Apply (16) by including in the initial test 7 only those items for 
which D;e > 0.52 WN, where z is the normal deviate corresponding to а 
chosen level of significance. 


z =< ——— 
————— MÀ —— 
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4. Score test T (with items scored in the directions determined in Step 2) 
and mark the scores on the answer sheets. Tally 7 and obtain its standard 
deviation S; . С 

5. Separate the answer sheets into five piles as in Step 1, but this time 
according to their T' scores. А 

6. Write in the frequencies for the cells of the 5 X 5 contingency table 
for the transformed T-scores and C-scores. Compute >>” T'C', first noting 
that scores falling in the most extreme categories on either variable receive 
values + 2 or — 2, while those in next most extreme categories receive values 
+ lor — 1. This can be done in a few minutes by counting, and frequencies 
for the center categories can be ignored. А 

7. Obtain the D;; from item counts as in Step 2. Check for retention 
in the test any items for which D;; < 0.125N/58; . It is possible there may 
be no such items. ; 

8. Compute the constant which is the left side of (26); set up the right 
Side of (26) for each item j (the operations may not have to be carried out) 
апа retain only those items for which (26) fails to hold. 

9. Score the test M, comprised of items retained after Step 8. Tally M 
and obtain its standard deviation Sy , or use Sy = mSz/n. uu" 

10. Separate the answer sheets into five piles as in Step 1, this time 
according to the M scores. е 

11. Obtain D; from item counts as in Step 3, but only for those items 
previously discarded in Step 8. 

12. Obtain bs M'O" as in Step 6 and compute the left side of (27). Set 
up the right side of (27) for each item for which D;ar was obtained in Step 11, 
and mark items for which (27) holds to be put back into test M. This completes 
the first cycle of the iteration, and a large proportion of the possible increase 
in validity will have been obtained. : г 

18. For convenience, again call the test obtained after Step St io 
T,” and repeat the procedure starting with Step 5. The паа КА 
at a point where either (26) ог (27), applied alternately, wi р e Hed 
further increase in test validity. Always apply (27) to a REA tes Y (28) a 
items. If S’ is a transformed score on the final test, 2; P /. ШР * 

(27)] is a conservative estimate of the final validity coeficient. 


IV. The Problem of Test Н omogeneity 


; i sistenc, 

A question which naturally arises 18 how Py e pep S 
tests will have when constructed by the method. fud with higher test 
Shorter and at the same time some of the pedona : bet It will usually 
correlations are dropped, its homogeneity тау в 1 test is long enough. 
remain relatively high, however, especially if the ini e be high (in practice 
Aside from length, another reason why homogeneity wi ) is that initial test 
between .82 and .90 for final tests of about 100 items 
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items are limited by (16) to those having significant criterion covariances; 
this condition alone tends to select items which correlate positively with the 
total test and thus enhance homogeneity. 

Cronbach (2) has demonstrated the relative importance of high item-test 
correlations and test length in contributing to homogeneity. In symbolism 
of the present paper, 


7, 
Şe > (= - 5) n», (28) 
V; n 


is а close approximation to Тете? (тар) > Ттт Where the latter аге K.R. 20 
coefficients for test Т, including and excluding item j, respectively. Inequality 
(28) can be used to determine how large the item-test covariance should be 

i е to the homogeneity of Т. When (28) is trans- 
formed as in Part II, it is fo 


à , und, for tests and samples of only moderate size, 
that all items for which D;r > 1 will contribute to homogeneity. 


V. Applications and Discussion 


applied using г = 1.96 (see Part III, St 


ms put one of them back, giving a 66- 


-65. Subsequent applicat; 27 
resulted in rejections and selections of from 1 nue wai 
accompanied by slight decreases in validit; i 

y. The 66- e 
accepted as the final test, for this sample, item test, was therefor 


value undoubtedly largely spu 
lations and because of the sma 
items to increase validity to .8 


1 criterion corre- 
ll sample size. Application of (26) discarded 9 
0, and an application of (27) put 6 items back 


кыа Эр" 
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into the test to make a 38-item test with validity of .82. This was not exceeded 
by subsequent iterations. 

"a In several applications the necessity for using only items which were 
individually valid was demonstrated. In one case all positively scored items 
were used in the initial test, that is, condition (16) was not applied. The 
Convergence was slow, and because of the presence of items which would be 
invalid in subsequent samples, the gain in test validity would not be expected 
to be permanent (see Part II). In another case only 58 items out of 677 
available could be found which were related at the .05 level to grades of college 
freshmen. About 34 items would therefore be expected to have only a chance 
relationship to grades. Application of the method retained 41 items and 
raised validity from .52 to .60. But in a subsequent sample the validity was 
pom as small for the shortened test (.22) as for the initial test (.16). The 
(A age in both cases was obviously due to the large proportion of invalid 
uM ae the initial 58. These results show that it is necessary to insure 
Pw m id ity in the initial test before applying the rest of the method; this 
ls y e achieved either by applying (16) with z large when there is only one 

rge sample, or by using several samples if z must be smaller. 

For p of the items are valid, the method appears to be worth applying. 
eL e, in а study reported elsewhere 178 items were found each to 
D. sis e ae the .01 level with a criterion. The test validity was raised from 
(5) еі A in the first sample (V = 441) by applying Flanagan’s method 
ont $4 is an approximation of the present method. A year later the 
this Nu test. (124 items) correlated .74 in а new sample (QN — 402). In 
f тазе the difference .74 — .66 is significant, using the traditional z-trans- 
ormation test, at the .03 level. Despite this apparently permanent gain in 
validity , the merit of selecting items for validity cannot be finally assessed 
until the appropriate sampling statistics are derived and applied. 
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WEN, ee to which scale values computed by the method of successive 
u M iverge from theoretically "true" values is seen to be due to three 
SI ие р due to inequalities in variances of the distributions from 
Ета scale values аге computed, error due to non-normality of the 
tou cl ions, and sampling error. The contribution of each type of error to the 
priate or а evaluated; the latter is seen to be surprisingly small under appro- 
P conditions. Certain aspects of the formal methodology underlying 
ing procedures are also briefly considered. ` 


in н apri popular and perhaps the simplest of all methods by 
shite taa : ies e assigned values for some psychological variable is the 
we n Me nique. Basically, а rating scale is some set of categories that 
ula "might ы que into mutually exclusive classes. For example, a rating 
phos e defined by the categories high, medium, and low, and а set 
i hy jud. generated by “the evaluation of the esthetic value of art object 
judges es i where i and i range over specified classes of art objects and 
ratin ^ ш is, each judge j assigns each art object 7 to a category of the 
ee Scale, such an assignment constituting an event. Corresponding to 
E event designated by the coordinate pair ?, J there is one and only one 
ategory: high, medium, or low. 

я Usually, the localization of each event on 4 
epresentation of various subclasses of the events by а single value of the 
scale. This can be done by taking the most representative scale value of the 
distribution of scores in a subclass as the scale value for the subclass as à 
whole. Thus, for the example already given, we may be less concerned with 
the rating given to a particular art object by а given judge than we are In а 
rating representative of the values assigned to that object by the various 
judges. Since a specific art object defines а subclass of ratings, the most 
representative rating (however defined) can be taken as the value of this 


stimulus on the esthetic scale, not dependent upon any particular judge. — 
i le imparts no measure of quantity 


sed of a set of mutually exclusive 


the scale is only а means toa 


ana *This paper reports research undertaken i 
nd Container Institute for the Armed Forces, E н 
вегіев of papers approved for publication. 'The views or conclusions cont: 
are those of the authors. They are not to be construed as necessarily 
indorsement of the Department of Defense. 


165 


166 PSYCHOMETRIKA 


categories. An example of this simplest type is the standard color chart by 
which colors are classified. Stevens (9) has named this kind of scale a nominal 
scale. With appropriate additional data, assumptions, or definitions, however, 
the rating scale can be utilized as an ordinal, or even an interval scale. 

If a relation can be obtained which orders the categories, then the rating 
scale has become an ordinal scale for that relation, One of the more customary 
ordering relations employed by psychologists in generating ordinal scales is 
that of preference, or choice. If the categories are such that, for a given 
judge, (7) an item assigned to category A is always chosen over any item 
assigned to category B (at least at the time of the assignment) and any 
item assigned to category B is always chosen over an item assigned to category 
C, and (ii) no item assigned to C is ever chosen over items assigned to А, 
then categories А, B, and C are ordered by the relation of preference. 

It is customary at this point either to define or to hypothesize the 
existence of a psychological continuum underlying the categories of the 
rating scale, such that each category covers a range of the continuum, these 
ranges being exhaustive, mutually exclusive, and in the same ordinal relation 
as the corresponding categories. In short, the rating scale is interpreted as 
a gross technique by which the values of events are estimated on a similar, 
but much more discriminating underlying scale, Thus, art objects evaluated 
їп terms of a three-category scale are assumed to be much more finely dis- 


FIGURE 1 
the Observed Rating Capone "Sone ioo ап Interval боца Seale С Represents 
Scale B, the Metric (with Arbitrary Origin and Unit Distance) Assigned oN he с. а" 

tinguishable esthetically. This is illustrated 
of the categories C, , C; , C, . Events fallin 
higher or lower than events in C;(i + j) 
no distinction is made among the events fall 
which is hypothesized or defined to unde 


in Figure 1. Scale C is comprised 
Б In a category C; are ordered as 
for the property being rated, but 
ling in C; . Scale cis the continuum 
tlie scale C, the smaller categories 


AN 
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indicating much finer differences in degree of the rated property. Strictly 
speaking, the underlying scale need not be an actual continuum; it may be 
conceptualized as a finite number of subdivisions of each category of the 
rating scale, as long as these subcategories are ordered by the same relation 
that orders the rating scale. 

If numbers can now be assigned to the various positions on the under- 
lying scale in such a manner that an interpretation can be defined, discovered, 
or assumed for the relative differences between positions on the scale, the 
theoretical underlying scale becomes an interval scale indieated in Figure 
1 by scale B. Then the assignment of an event to а category of C is interpreted 
as an estimation of the score of the event on the underlying scale B. We shall 
refer to theoretical metrics such as B, defined or inferred from cruder empirical 
measures, as base scales. It has been cutomary to define the base scale (more 
rigorously, a set of base scales, linear transformations of one another) for a 
particular set of categories and distributions over the categories as that 
assignment of numbers to the continuum which normalizes the distributions 
(10). There may be other equally valid ways of defining the base scale, e.g., 
the counting of just-noticeable-differences, Or defining the base scale so as 
to normalize distributions other than the ones being dealt with in the given 
study. It is not always possible, given more than one distribution over the 
same continuum, to find a numbering of the continuum that simultaneously 
normalizes all the distributions. Although the method of successive intervals, 
às described in the literature, has assumed normality for all distributions 
used in the analysis, we shall demonstrate that the validity of the method as 
a computational technique need not assume normal distributions. 1 

Опсе the existence of а base scale has been defined over the categories 
of a rating scale, each event classified by the scale is considered to have a 
value on the base scale. Since each category corresponds to an interval of the 
base scale, the assignment of an event to a specific category determines а 
range in which its base scale value falls. If, now, the shape of a distribution 


of scores on the base scale is known, values for the widths of the various 


category intervals can be computed in terms of the standard deviation of 
computed by tabu- 


that distribution as the unit of measurement. These are у 
lating the cumulative proportion of scores at each boundary of the ds 
апа caleulating the width in sigmas corresponding to such а percentile 
difference for that type of distribution. If the distribution 1$ known or 
assumed to be normal, then the interval width will be the standard deviation 
of the distribution multiplied by the difference between the normal deviates 
Corresponding to the cumulative proportions at the lower and upper 
boundaries of the interval. If a number of distributions are available, Xe. 
a group of judges rates a set of stimuli so that each stimulus determines : 
class of ratings, a number of measures of each interval width will be Ба 
in terms of the sigmas of the various distributions. If these are pooled, 
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estimates of the interval widths in terms of a common unit of measurement 
are obtained. Finally, if the median of a distribution be taken as the base 
scale score representing the distribution, the exact base scale value of this 


multiply this proportion by the interval width, and add the product to the 


conversion of distributions of scores of 


to points along an interval scale have been 
variously described in the literature, 


Previous advocates of the method of Successi 


1 ve intervals have attempted 
to validate the technique by d 


emonstrating its extremely high correlation 
parisions (8), and its internal consistency (3). 

in terms of the degree to which 
results of the com i iri ata can be expected to diverge 
values as determined from the definition of the 
absolute validity of the method. 
Ч by the method of successive 
vals relative to some arbitrary 


: idths, since once the latter are 
known the former i 


nown t) - It is obvious that (7) if all the 
distributions used to measure the interya] widths have equal variances, (0) 


) if there are no sampling errors, 


A rval widths are identical with the 
theoretical values. 
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of all the distributions were equal, then each distribution would give the 
same computed value for a given interval width. Thus there are three possible 
sources of error in computation of base scale values by means of the method 
of suecessive intervals: type (а) errors due to unequal variances of the 
distributions used to compute the interval widths, type (b) errors due to 
non-normality of the distributions, and type (c) sampling errors, i.e., errors 
due to the estimation of cumulative proportions of the interval boundaries 
from finite samples of the measuring distributions. 

As a tool for evaluation of the contributions of these sources of error 
to a total error of estimate of relative interval width, it is convenient to 
define a. coefficient of error. Let & quantity x be estimated by a quantity X. 
Then the coefficient of error, £, for the estimation of x by X is = (X — 30x 
or X = (1 + 4x. The magnitude of the coefficient of error gives the dis- 
crepancy between X and x as a proportion of x and is nothing more than 
1/100 of the percentage error in the approximation of x by X. 

Relative interval widths computed by the method of successive intervals 
are estimates of true relative interval widths on the base scale. By relative 
interval widths, we recognize that the unit of measurement is arbitrary, SO 
that the ratio of one interval width to another (which is invariant under 
transformation of the unit of measurement) is the critical quantity by which 
relative interval width is expressed. We can evaluate the coefficient of error 
for the estimation of true interval width ratios from computed ratios as 
follows: (i) find an expression for the computed interval widths, L; and Lr , 
Tor categories j and k, in terms of the three types of errors that influence the 
computed widths, and of the true interval widths, A; and № ; (її) set 

L;/L; = 1+ £j)0,/N)- a) 
Solving for £j, , we obtain the coefficient of error for the estimation of relative 
interval widths by the method of successive intervals as а function of the 
different types of error; we shall be able to see explicitly the manner and 
extent to which each kind of error contributes to the total error. 

Let A; be the true width on the base scale of an interval j in terms of 
Some arbitrary unit of measurement U, and L; the width of the interval as 
computed by the method of successive intervals. Let л; (measured in terms 
of U) be the standard deviation of the ith distribution over the base scale. 
If the ith distribution is normal and displays no sampling errors, then the 
cumulative proportions at the upper and lower boundaries of sc 2 
Permit an exact computation (through use of a table of the normal proability 
integral) of the magnitude of А; in terms of n; 25 à unit of measurement. 
Specifically, 

l; = N/m = и ) 
Where 1;; is the width of interval j as computed from distribution $ N and 
п; are the true magnitudes of interval j and the standard deviation of distri- 
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bution 1, respectively, in terms of the arbitrary unit of measurement U; and 
>; = 1/n; . However, to the extent that distribution 7 diverges from normality 
and contains sampling errors, l; will differ from ¥:A; . In general, l; = 
vA; + f, where fi; is the discrepancy between the computed /,; and the 
theoretical v;A; . $i; can be analyzed into two additive components E,; and 
€i; , Where E;; is a constant bias due to non-normality of the distribution and 
€i; is a random sampling error, Thus, 


li; = vid; T E; + um (2) 


It should be noted that the unit of measur : 
©; is n; , the standard deviation of the ith distribution: while the unit of 


2 


measurement for 1/>; and А, is the arbitrary U. Which is the same for all 
distributions. 


The computed width L; of interval jisa weighted average of the estimates 
of widths contributed by the various distributions, That is, 


І, = >; wil; = À; Ж Wi, 4 D WE; 4 5» Ш; ;є;; у (3) 


where У), Wi; = 1. Defining the quantities А, » B; , and y; by 


мі: Eu , (4) 
В, = (У wE Ад), ш 
з = (EX ANAA), © 
we obtain | 
= AK + 8, 4), ү 


that 1/4, is approximately the harmoni 


We are now able to evaluate the co, 
ratio L;/L, as an estimate of the true 
j and k. Finding L, by substitution o 
for £j, in (1) we find that 


efficient of error, £;, , for the computed 
ratio X,/A, of the widths or intervals 
f k for 7 throughout (7) and solving 


— € 
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which may be written 


1-8.) В; — Bi Yi — Yk 
е «ваи l- 4e. RET (8) 


where 
Qik = (А;/ А) — 1. (9) 


Since o;, reflects the difference between units of measurement within interval 
j and within interval К, and vanishes (as shown below) when the variances 
of all the distributions are equal, a;, may be regarded as the error in relative 
interval width due to unequal variances of the distributions which are used 
in estimating the interval widths. Thus, of the three sources of error in the 
method of successive intervals, type (a) is represented quantitatively by 
a, type (b) by 8, and type (c) by v. 
y-Error 


It will be recalled from (3) that each distribution 7 was assigned a weight 
W;; for its contribution l:; in the computation of L; . It is now possible to 
assign these weights in а manner that minimizes the sampling error y; . 
Assuming the various distributions to be essentially independent of one 
another in their sampling errors, we find from (6) the mean and variance 
for y; (under repeated sampling with a fixed set of weights) to be 


bu = Y шаша ЈА ; (10) 


Y utet КАА). (п) 
апа à;,, are the sampling errors for the 
standardized deviates of the normal probability distribution correponding 
to the cumulative proportions of distribution 2 at the upper хан the loy z 
boundaries of interval j, and ô = A P/y, where AP ЕТЕ “ч a oi 
a cumulative proportion at an interval boundary an ni tl E 08 шы) S 
the normal probability distribution at P. Since the sampling m 

Zero, ue; œ 0 and hence, from (10), 


\ 


2 
Oyi 


But e; = ày,, — бу » Where би; 


~ (12) 


py; 0; 


While 
CHE m Tiv + 08,41 — 2 соу (би; › 8) е (13) 
[ = Pow) ү РАКЕ Pu) 2Р1 — Pu) au) |, 


F yu Ура ШАД 
Ww) E ion î, Р i tric 
4 i istribution 7, Pr (Py,,) is the parame 
here n; is the sample size of distri pum E DA TURON n dar 


cumulative proportion of distribution 7 
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of interval j, and y;,, (yv,;) is the ordinate of the normal probability distri- 
bution at P;,, (Py,;). Since the sample size n; is known, and the sample 
cumulative proportions provide sufficiently close approximations to the 
parametric cumulative proportions, very close approximations of o7,, may 
be computed from empirical data by use of formula (13). Furthermore, 
c,,, may be made as small as desired by choosing the sample size n; suf- 
ficiently large. 

The assumption of (11) might at first seem gratuitous; for many ex- 
perimental situations the sampling errors of one distribution will not be 
strictly independent of the next. "Thus, if two sample distributions are obtained 


non-linear correlations exist; and (їй) the i 
positive and negative values, so that eve 
are significant their net effect may be negligible. Thu 


Since by (12) the average value of y, 
absolute magnitude of Y; is less than (t 
expected (absolute) size of y; will be 


: 2 : 
5 Vig; = k; , where k; is a constant 


4, wt of proportionality. Since У. Wi = 
1, k; > ( i к) ‚ SO 


W; = (етт = Secs) > (14) 


should be acceptable for most Purposes.) 
Substitution of (14) in (11) gives 


95, = (4 (57 а). (15) 


Since o7,, is usually on the order of l/n,, letting n be the average size of the 
sample distributions and N the number of distributions, c? is roughly on 


TT 
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the order of (А;А;)° (nN). Thus the expected order of magnitude for 
y; is roughly (4,3) * (nN )*. This value may be made as small as desired 
by taking sufficiently large n and N. For example, if A; = .5, N = 50 and 
n = 500, then the expected order of magnitude for 7; is 107°. Thus for an 
empirical study of any substantial proportions an expected order of magnitude 
for y; of 107° should not be difficult to obtain. In order to maintain a fixed 
order of magnitude for ү; , a decrease of interval widths must be compensated 
for by an increase in (i) the sample sizes, (#2) the number of distributions, or 
(iii) both. For with o,, held constant, VN is inversely proportional to 
A;\; , while the latter, as shown below, is the width of category j in units 
of measurement given by the harmonic mean of the standard deviations 
of the measuring distributions. This has direct’ implications for the design 
of rating scales, for it shows that the number of categories into which a 
Scale can be reliably decomposed is limited by the number of stimuli and the 
size of the population upon which the scale is to be standardized. 

... Thus, if the width of an interval relative to 1/4; is not too small, and 
if the study by which the scale is being standardized is of reasonably sub- 
stantial dimensions, the error in estimation of A; due to sampling will be 
insignificant—generally on the order of 1072. In light of this, the mean and 
sampling error of £,, can be evaluated. Any reciprocal, 1 /5; , from a distribution 
ОЁ s with mean M, can be replaced by the expression (2/M, — s,/M;) with 
an error coefficient of — [(s; — М.)/М.}. Since from (12), the mean of 
1+6+y)is а + 8, Ша + & + 3;)] may be replaced by [1 + 8. — 
т)/ (1 + B,)°], with an error coefficient of — Г) + 8), the error of the 
replacement being negligible so long as 8. does not approach — 1. With this 
replacement we find from (8) and (12) that the sampling mean of £j, is 


ща tx aall + 8)/(1 + 82] + (6; — В2/(% + 82, 
While, disregarding second-order terms, 
eu, се [(1 + os) / + BIW В) + б + By тү = 


(16) 


(17) 


a-Error 


For evaluation of еу. , the error due to inequality of variances, it is 


Convenient to employ the identity 


N 
Ag = p» ш>; = Ме„;Ф,Гең› ONUS 
i=l 
= Nosta +7; 
is the standard deviation 
utions, c, is the standard 
is the product-moment 


Where N is the total number of distributions, 9v; 
с the weights for the jth interval over the N distrib 
eviation of the v, over the N distributions, ?v;» 
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correlation between w;; and >; over the № distributions, and 7 is the mean 
value of v; over the N distributions. From (9), this gives for a;, 


Nosite — Nono, 


wer 


Маза oar. ЧЕР (18) 


a (See ЕЕЕ бу 
Crew, +1 AP 
where C; = Noy, , С, = Ncs, , and V, = c,/ V. А 

The value of C; depends only on the shape of the distribution of weights 
for the interval j; this value will be of an order higher than 10^* only when а 
relatively small proportion of the distributions receive a significant weight 
for interval j. In the case where a proportion, К, of the distributions receive 
equal weights and the rest receive 0 weight, C = V(1/k) — 1, which exceeds 
1 only when k « .5 and is no larger than 3 when k = .1. The correlation, 
T«;, , between the weights assigned to the distributions for interval j and 
the reciprocals of the standard deviations of the distributions can be expected 
to assume some small negative value (with a chance divergence which 
vanishes as N grows large), since as у, increases the boundaries of the interval 
draw closer to the center of the distribution, yielding an increase in w,; . How- 
ever, we should expect this correlation to be equal for both intervals j and K. 
Thus, the maximum value of а; would be approximately V, X 107’. 

But V, is the coefficient of variation for the reciprocals of the standard 
deviations of the measuring distributi 
the coefficient of variation for the s, 
on the order of 1071, which makes o 


Gu = 


les in variance suggested by Attneave (D- 
It should be noted that if all distributions receive equal weights for two 


az = 0, regardless of the magnitude of V, . Even 


› 2 will be negligible if the ш and w,, are sufficiently 
homogeneous. It should also be noted that 


А; = (Cr, V, + 1)рлур, (19) 


n of the л; . This substantiate 
interval widths are expressed 11 


B-Error 


Of the three sources of error in the method of successive intervals, evalua- 
tion of 6-error is the most difficult. We can replace Уу, w,E,;, by (News 
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ств; + Ej), where E; and тк, are the mean and standard deviation of the 
errors introduced into the estimation of A; by the non-normality of the N 
distributions. Then from (5) 


B; = (Совт: + EDAN). , (20) 


Note that Ё;; is measured in terms of the standard deviation, 7; , of 
distribution 7. In particular, when Pz:; and Py,, are the cumulative pro- 
portions of distribution ? at the lower and upper boundaries of interval j, 
E, is the difference between the number of sigmas spanned between BL 
and Py,, by the actual distribution i and the number of sigmas spanned 
between P,,, and Py,; by а normal distribution. Let D;; be the number of 
sigmas spanned by distribution į between these two cumulative proportions, 
and let d,; be the corresponding number of sigmas spanned by à normal 
distribution. Then E; = d; — Ри = Ри , Where ou = (du/Ds) — 1 
and is thus the coefficient of error for the approximation of the distance in 
sigmas spanned between Py, and Pz:; by distribution i by the corresponding 
distance spanned by a normal distribution. Since Di; = N/m: = "iN; (20) 


may be rewritten as 
B; = (Oif oiz: umi + op) / CAN) 
= Отвор F o,9/ A2- 
But when V, is small, 


Ooi(/ai) 2 Oui 


and 
«;(v/ A) = АД té. 
Therefore, 
Bi & с„(Отьк + Viros) + ё (21) 
eite, 
Where 


В; з с. (Сз; + ль). (22) 


In general, while there may be some small non-linear correlation between 
as N increases and the 


ш; and E;; , the linear т„,Ё, will be close to zero. ub 
chance fluctuation of Ve;z; thus diminishes. А similar argument holds ie 
Ta» ; because of the small expected values of C; and V, , 8; should be on the 
order of c,, X 107* at maximum. It will be shown below that even E a 
distribution is markedly non-normal, the expected order of magnitude 2 
9; is only 107, so cw; Will be on the order of 107' at maximum. Thus, 6; 9 

be on the order of 107° at maximum and is more likely to be of order 107. 
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It follows that the only 


latter comprising the average 
measure the width of interval j. These № 


2 


a; , and variance, cb; , Of 
pulation of distributions 
› He; , and variance, оз. Finally, since à; is the 
mean of a sample of size N from the w; , дь, = Hu; and o5; = c»;/ N; - 
(More generally, o2,/N < 95; < бї, depending upon the extent to which 
the w,; for the sample of N distributions are independent of one another. In 

ituati at the w;; are not wholly independent, 
justify equation (11), we shall expect 
e negligible.) Let /’ be the extent to 


y= BY р, 23) 
and thus, from (21), PARE Fay à: 
В = BER By uu. (29 
Since 8’ is of order т„,/ VN, and as already mentioned, the expected order 
of magnitude for о, is 19-: or smaller, then if N is reasonably large the maxi- 
mum expected order of magnitude for 8/' is 107. This leaves Hw, in (21) as the 
only component of 8; likely to be significant. But Ha; is merely the expected 
value of w;; on the interval J. As indicated below, the absolute magnitude of 
i 15 only of expected order 10™ even when the distributions are quite non- 
normal. While it is impossible to make any definite statement about the 
average, uu; , for an interval J, it would seem unlikely that it could exceed 
-10 except under cases of extreme, Persistent, and Positively correlated 
non-normalities among the Population of distributions over interval j. Thus, 
except under unusual circumstances, B; is of expected order of magnitude 
10" or less, and we may simplify (16) and (17) to 


Ша San + В, — В, 


(25) 
Serer (Bj = В) ү (Bi — вр) 4 Ho; — Mar 
Tia TW e»; T LM 29 


Of the terms in (25), only rm 


and 


гапа ua, are of expected order larger than 1077. 


Ө the anticipated order of magnitude for c. 


reasonably may be anticipated to fall and hence obtain reasonable bounds 


байле М 
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н 
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for the magnitude of w. What we shall illustrate here is a technique by which 
a distribution of any given shape readily may be inspected for its values of 
w. By this technique, the reader may select what he considers to be fair 
examples of empirically anticipated non-normal distributions and easily 
convince himself that о is unlikely to be of an order greater than 1071. 

It will be recalled that o — (d/D) — 1, where d and D are the distances, 
measured in terms of the standard deviations of the distributions, spanned 
between the cumulative frequencies at the upper and lower boundaries of 
the interval by a normal distribution and the empirical distribution, re- 
spectively. Let Py and Р, be the cumulative proportions at the upper and 
lower boundaries of the interval, let у(х) and Y(a) be the ordinates at x of 
the unit normal distribution and of the empirical distribution standardized 
to с = 1, respectively, and let хь and X; be the distance of cumulative 
Proportion P from the means of the unit normal distribution and the standard- 
ized empirical distribution, respectively. Then d = ze; — Tr, and D — 
X Py = Xp». Büt 
IP 


Pu - Pi = | 


where ў is the mean value of the ordinate to the unit normal distribution 
over the interval. Similarly, 


" y(a) dx = jd, 


PL 


Xpy < 
Ре == Py LA Yla) de = YD. 


Hence d/D = Y/j, sow = (Ў/ў) — 1 and is thus the coefficient of error for 
the approximation of the average height of the unit normal distribution 
between two cumulative proportions by the corresponding average height 
of the standardized empirical distribution. The magnitude of w is then 
readily seen by an inspection of the graphs of y and Y against P. That is, 
let (P) = y(xp) and Y(P) = Y(X»). It is computed without difficulty that 
U(x) = g(P), where j(P) is the harmonic mean of y(P) between Pz and Py , 
and similarly Y(X) = Y(P). Also, except for those intervals over which 
the coefficient of variation for y(P) or Y (P) is large, g(P) = g(P) and Y (P) = 
Ү(Р). "Thus, given any empirical distribution, the magnitude of the approxi- 
mation error can be determined readily by standardizing the distribution to 
Unit variance, graphing the height of the distribution against cumulative 
Proportion, and superimposing the corresponding graph of the unit normal 
distribution. One may then select two cumulative proportions, estimate the 
Average difference between the curves over the interval visually, and divide 
this by the estimated average ordinate of the normal distribution over the 
interval. 

We illustrate the method through its applica 
distributions, a rectangular distribution and а triangu 


tion to two arbitrary 
lar distribution skewed. 
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so that the projection of the apex divides the base in a ratio of D 
are shown with unit variance in Figure 2, together with the ws um 
distribution by which they are to be approximated. Both prec 
represent departures from the normal that, in an empirical is 2 ccm 
would be considered severe. Figure 3 shows the same distributions i 


50; 


~ Rectongular 
~: Triongular 


а, ee ел, oe Normal 
40 : 5 


зо 


<-— 


20 


Normal, Rectangular, and Triangular Probabilit; Distrib 
Projection of Triangular Apex Divides the В, 


arious pairs of cumulative Proportions are selected and € 
estimated, it is seen that | e | has a mod 
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the modes are well separated and the intervening troughs deep) the severe 
non-normality of the distribution should be painfully apparent when the 
distribution is plotted on the successive intervals scale as finally computed. 
The non-normal distribution then may be discarded and a new analysis of the 
remaining data performed if the investigator sees fit. 


Rectangular 


.50 
—-—-— Triangular 


———-— Normal 


40) 


20) 


80 .90 1,00 


em 1077 7120777 1807 40:7" 30503 Poo STO 
P — 
FIGURE 3 


Ordinates of the Distributions of Figure 2 as Function of Cumulative Proportions. 


B-Error and Its Relation to the Base Scale 


und it unnecessary to make any comments concerning 


So far we have fo 
e rating scale except to hypo- 


the base scale which supposedly underlies th 


thesize its existence. 
Necessary and sufficient conditions for the existence of an interval base 


Scale underlying a successive interval rating scale are: (2) There must 
(potentially) exist a numbering of all the potentially infinite number of events 
classifiable by the rating scale such that there is no overlap, for any two 
categories of the rating scale, of the ranges of the numbers corresponding to 
the events falling within each category. (ii) The positions of the ranges 
Corresponding to the various categories must be in the same ordinal relation 
as are the categories. (iii) For any веб of events so numbered, there must 
exist some interpretation of (a) the ordinal relations among the numbers 


180 PSYCHOMETRIKA 


е п 
assigned to members of the set and of (b) the ratio of the wx ae e 
the numbers of any pair of the set to the difference between the = at thd 
any other pair, in terms of some properties of the events of the = a hee 
base scale provides interpretations for additional properties of t s S Ther 
assigned to events, it may become a ratio or even an absolute e A i 
may be many different numberings satisfying conditions @ an i Nec 
many different interpretations in accordance with condition m recs 
there may be many different base scales underlying a given successive it ү; 
scale. In fact, any assignment of numbers satisfying conditions е ^" 
is а potential base scale for the rating scale since we can never kno 


rating scale category. In particular, 
a potential base scale is also a potential base scale. 


a is is the 
The essential result of a method of successive intervals analysis is 


i. Р Р zals 
derivation of a set of numbers corresponding to the boundaries of theinn 
ers, when paired and the ratio of differet 


base scales minimizing the En. 


Since the exact magnitudes of the 


ыш е 
£j, are unknown in applications of th 
method of successive intervals, it is 


impossible to determine the class i 
ely approximated in a specific instance. T > 
most likely to minimize the £j, , however, n 
fed values of the £., | Now, the data of a speci ^ 
successive intervals analysis are obtained by sampling of two kinds: a samp 

of size N from possible distributions over the rating scale, and a sample v 
n; individuals from each distribution ¿ (; = 1, 2, ... , N). The pore 
value of £;, for a specific sample of distributions is given by (25). But pa 
terms оь, (8; — В), Bi’ , BY’ are dependent upon the specific sample 
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distributions chosen; 8/^ and 87 , by definition, have an expected value of 
0, while both a;, and (8; — 8t) are determined essentially by differences of 
the form X,7r,,.; — Хг, where X, y, and z are various specified properties 
of the N distributions. These differences should be negative as often as positive, 
so that the expected values of а; and (8: — 8) should be 0. 'Thus, the ex- 
pected value of &;, is approximately ua; — Her › and hence the classes of 
potential base scales most closely approximated by the expected computed 
scale are those base scales showing the smallest differences among the ды » 
He, , +++ for the various intervals j, k, ++ of the scale. This important 
conclusion may be rephrased as: the classes of potential base scales expected 
lo be most closely approximated by the method of successive intervals are the 
classes for which the average coefficient of error (for the estimation of interval 
widths under assumptions of normality) is most nearly the same for all intervals 
of the rating scale. 

In particular, if, as implicitly assumed by previous psychometric analyses 
wherein the base scale remained unidentified, there exists a class of potential 
base scales which simultaneously normalize all distributions, then р, = 0 
for all intervals of these scales; there is no class of base scales more closely 
approximated by the expected computed scale. 

Thus, we see that there is no single answer to the question of the 
magnitude of error involved in the approximation of an unidentified base 
scale by the method of successive intervals; the magnitude of error is relative 
to that base scale for which the computed scale is considered an approxi- 
mation. If we wish, however, we may define the base scale to be approximated 
as that scale which simultaneously equalizes the pa for all intervals. A class 
of such scales can always be found, and further, the set of all such classes 
includes all base scales which simultaneously normalize all distributions over 
the rating scale if such scales exist. If the base scale is so defined, then from (25) 

ща = an + (8; — BD + (87 — 8i). (27) 
Only when the measuring distributions are extraordinarily non-normal are 
any of the terms on the right side of (27) of expected magnitude greater 
than 107°, and thus £;, has an expected order of magnitude of no greater 
than 107°, This, in conjunction with (26), shows that if the sample sizes of 
the distributions have been taken sufficiently large (say, large enough to 
make c, on the order of 107°), then the extent to which interval ratios computed 
by the method of successive intervals diverge from the corresponding theoretically 
“true” values should not exceed 10 per cent of the latter, and may be much smaller 
if the experimental study has been well designed. 


Conclusions ү « 
Abstracting the essentials of the foregoing analysis, three major points 
are of significance—the first, a contribution to the computation Non 
of the method of successive intervals; the second, an evaluation of the validity 
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of the method; and the third, the significance of the method for the basic 

methodology of psychophysical measurement. | i3 
The contribution to computational technique is given by equations ( 

and (14); it involves the computation of weights for the estimates of a given 


si 
or unless suitable tables have been obtained, the improvement of this exact 


sampling error. | 3 
The validity and reliability of the method of successive intervals 
not depend upon normality of distributions or equality of their variances. 


The reliability, as attested by (26), may be made as high as desired. If the 
base scale is suitably defined (ie., d 


? 
intervals, the error due to estimatio i i 

normal probability integral) and if the reli 
then the validity, as implied by (27), 
coefficient of error for relative interval 


Finally, and probably most important of all 
of this analysis for the methodology of 
been shown i 


| 
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then (b) that scale, or another very similar to it, has interpretive significance 
as an interval scale. The latter premise is both weaker and stronger than the 
former: weaker in that a scale satisfying (a^) can always be found, and such а 
scale also satisfies (а) when scales satisfying (a) exist; stronger in that the 
latter premise demands a meaningful scale to underlie every psychophysical 
measuring technique, whereas the former demands such a meaningful under- 
structure only if a psychometric scale can be found to normalize simul- 
taneously all distributions over it. Actually, the (b) clause of these premises 
is not so strong as it might appear. In a certain sense, the mere act of defining 
a scale in terms of the distributions over it imparts a meaning to the scale 
values so defined. Essentially, what our present analysis has shown is that 
it is always possible to give a distributional definition to a base scale which 
simultaneously normalizes all distributions regardless of whether or not а 
Scale exists. 

Since interpretation of psychometric scales has been sought in actual 
Practice, regardless of whether simultaneous normalization could be realized, 
it is essential, if psychometric custom now current is to be justified, that a 
way be found to define psychometric scales in terms of properties other 
than such normalization. It is our belief that such justification has now been 
furnished. 
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RELATIONSHIPS BETWEEN TWO SYSTEMS OF 
FACTOR ANALYSIS 


C. W. Harris 
UNIVERSITY OF WISCONSIN 


Considering only population values, it is shown that the complete set of 
h units in the diagonal cells may be trans- 


factors of a correlation matrix wit l 1 
formed into the factors derived by factoring these correlations with communal- 


One of the distinctions in factor analysis that may be made, when viewed 
at the procedural level, is the distinction between choosing to factor R, , the 
matrix of the intercorrelations of the variables with units in the diagonal 


cells, and choosing to factor R, the matrix with communalities in the diagonal 


cells, The purpose of this paper is to develop the transformation which relates 


the factors Г, of R, to the factors F, of R. In order to develop this relationship, 


it will be assumed first that the elements of R, and of R have been determined 


Without error, i.e., that the correlations and the communalities are population 


values. Guttman (2) has discussed conditions that are necessary for common- 
factor, or communality, solutions. Considering only population values, 
this paper shows that if a communality solution exists, it is simply a pro- 
jective transformation of the set of complete factors of Ri. 

The problem of estimation will next be considered. It will be shown 


that the transformation developed here when applied to observed data yields 
uirements for а maximum 


common factors that satisfy Lawley’s (5, 6) req 

likelihood solution and are a first approximation to factors derived by Rao's 

(7) canonical factor analysis. In developing these points, certain simplifi- 

cations have been introduced for convenience. For example, only orthogonal 

factor solutions are employed; however, such solutions may be rotated to 
variances and 


Oblique factors if such are desired. Second, the factoring of / : 
ly incidentally in connection 


covariances, rather than R, , is considered only ' " 
With Rao's canonical factor analysis. Third, it 1$ generally assumed that 


R, , and consequently F, , is nonsingular. This is а realistic assumption. If 
necessary it can be abandoned, provided we make certain modifications in 
the algebra; these modifications do not negate the generalizations given here. 

The distinction between choosing to factor R, and choosing to factor 
R is of considerable importance theoretically. In the former case the factors 
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are conceived to be located within the Space defined by the variables ies 
aS vectors, whereas in the latter case both the common and the bes 
factors are conceived to be located in a space distinct from the space c me 
by the variables viewed as vectors. Consequently, factoring R, is equi viai 
to developing both the factor coefficients and the factor scores by approp 


cannot be computed, even though the elements of R are determined without 
error, but must be estimated. 


product of two matrices, Y = FS, such that SS' is an identity matrix of the 


choosing the linear operators. 


Now let us examine the relationship between factors extracted from 
R, and those extracted from R, If both Z and ZA are given, we may write 


Z-HS, : (1) 


by which is implied that the matrix Z has been factor 


ed completely, usually 
with as many factors as variables; and 


2А -RS,, (2) 


with the implication that ZA has been factored completely, yielding m < w 
factors. It has been shown (3) that 525, defines A, which is the symmetrie 
idempotent matrix that achieves the projection of Z that is required by 


the communality principle. Therefore, since SaS = І of order m, we may 
write 


2815, = PS, , 
and 


283; =F, . 
Then the desired relationship is given by 
Р, = F (5,8%). (3) 


The matrix in parentheses in (3) will ordinarily be Singular; if so, it is not 
possible to solve (3) to write F,asa transformation of F,. 
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The matrix in parentheses in (3) is the transformation that relates the 
factors derived by the communality principle to those derived from the 
complete factoring of R, . That this is the relationship to be expected on 
other grounds is readily seen. Any matrix of unit-length factor scores, S, 
may be viewed as a matrix of direction cosines that gives the location of 
the factors with respect to the N person vectors, i.e., they locate the factors 
in the person space. With two sets of orthogonal factors located in the same 
person space, the transformation of the inner products of the variables with 
one set of factors to the inner products of the variables with the other set of 
factors is given by the correlations between the two sets of factors. It should 
be noted, however, that this is not a conventional orthogonal rotation; for 
example, the sums of squares of the entries in any column of the trans- 
formation need not be unity. 

A solution for the transformation T = (5,82) may be obtained. For 
example, 


(FiF) FIF; =T 


follows from (3). This is a true equation and not merely a least squares 
approximation under the conditions described here. This is to say that 
the symmetric idempotent matrix generated from F, is а unit for multiplica- 
tion of F, . If R, is nonsingular, then F, also is nonsingular, and so the solution 
for T might be written 


(F DF P Т. 
Another means of solving for T follows from the requirement that 


рур; = В = РТТ: = В, — U’ = ЁЁ! — 0, 


Where 17° designates the matrix of unique variances. Assuming Ё, nonsingular, 


а (F)?U*(F)". (4) 


A solution for Т (or some orthogonal rotation of T) is then given by factoring 
the matrix on the right of (4). At this point а solution for T is merely of 
theoretical interest, since if population values of correlations and communali- 
ties were known, the straightforward approach to determining the common 
factors would be to factor R. 


Estimation of Common Factors 


An important statistical problem faced by the factor analyst who 
le is that of estimation. Lawley 


wishes to employ the communality principle 18 t : 
(5, 6) has presented a maximum likelihood solution that has certain optimum 
int of view. He also presents a test of 


characteristics from the statistical po We d en 
significance for common factors derived by his iterative procedure. Rippe 
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(8) has extended the test of significance to factors derived by any method. 
Lawley's requirement on the factors F, is, in the notation employed here, 
DF; = FXU~R, — I) (5) 


, 


R, — R* may be made. 
Modify (5) by substituting FLT = р, ‚апа R, = pp: ; then with F, 
non-singular, 


DT’ = T'(rtU-p, — 2). (6) 


This states that to satisfy Lawley's requirement, each column of T must 
be proportional to a unit-length characteristic Vector of the matrix 
(FIU^F, — I). From (4) it is evident that a satisfactory solution for T is 
to choose each column as Proportional to a unit-length characteristic vector 
of the matrix on the right of (4), i.e., to define Т by a principal-axis factoring 
of this matrix, But this is merely restating the requirement of (6), since the 
characteristic vectors of (PIU? p, — Г ) and of [7 — Py) TUF) necessarily 
are the same. We have therefore shown that the Solution F, = F,T satisfies 
Lawley’s requirement when Т is defined by a pri 
matrix on the right of (4), 


Now it is at least intuitively evident that 
t A 


ncipal-axis factoring of the 


› NeW trial values of (2 must be employed, 


саре. missib] T and of 
the significance of the residuals must be made, шы. 


А relatively new attack on the proble 


modifies initial trial values of U* by an iteration Process, under the restriction 
of а given number of common factors. A test of Significance for a least number 
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of factors is provided. Rao also shows that canonical factor loadings derived 
from correlations are proportional to those derived from covariances, with 
the constants of proportionality given by the sample standard deviations 
of the variables. Therefore, in showing that the solution F,T = Р, is a first 
approximation to Rao's canonical factors, we imply that the solution based 
on variances and covariances, instead of R, , also is a first approximation to 
canonical factors. 

Rao's procedure requires that we select the non-zero elements of 0° 
to satisfy 


lA = Os — а, + б — Dal 4 E Om = Vain + 1, (7) 


Where each A; is a latent root of the matrix U^ F,F(U^, and each a;; the 
appropriate element of the unit-length characteristic vectors of the same 
matrix. Now these roots and vectors are connected with the roots and vectors 
of TT” in a regular manner. This connection is derived from the fact that the 
roots of 07:07! = (U^R,U^ — I) are all one less than the roots of 
Оу", and the characteristic vectors of 07:07" and U^ R,U'" are 
identical. Let L? designate the roots of U ^ F,FiU '; then the roots of TT’ 
are (I — 27°), Conversely, let the positive roots of TT’ be designated by 
Юл . Then (I — D2)^ yields the m roots in L^ that are each greater than 
unity ; call this matrix L2 . Let Q designate the unit-length rows of character- 
istic vectors of U~'F,F{U ~. Then the m unit-length rows of characteristic 
vectors of ТТ” coresponding to the positive roots Dj are given by TOU Er: 
It is now evident that P’D,, , defining m admissible columns of T, yields by 
the above transformations the required values for substitution in (7). Ap- 
parently, then, for any specified U? we may characterize the solution 
Р.Т = F, аз a first approximation to Rao's canonical factors. This is verified 
d noting that Rao defines canonical factors, at any stage of approximation, 
y 


Р, = UQ'(Lz — D^. (8) 


Substituting for P’ and Dn their expressions in terms of roots and vectors of 
U^ F,F(tU^'! and then simplifying gives 


ЕТ = FP'UD,-UQUb.— pa 


Which is identical with (8). 

This demonstration of the connection 
characterized as making explicit an alterna t 1 
Seems practically certain that Rao recognized the existence of this alternate 
path and rejected it for a very practical reason, namely, that his caleulation 
routine is less laborious than one based on finding F, by way of Ё, . 


of T with Rao’s procedure is best 
te path to canonical factors. It 
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TESTING MESSAGE DIFFUSION IN HARMONIC 
LOGISTIC CURVES* ) 


Srvanr Carrer Dopp 
UNIVERSITY OF WASHINGTON 


The growth of a population of knowers of a message was studied to test a 
human interactance hypothesis. The conditions investigated involved people 
interacting in time, with the population pairing off randomly (i.e., determined 
by many, small, different influences) and transferring an attribute (ie., an 
all-or-none act) at either a steady rate or a waning rate, subsequent to the 
originating stimulus. The mathematical expressions for these pre-conditions 
were the differential equations for the linear logistic for steady acting and the 
harmonic logistic for waning acting. Variant forms of these curves were devel- 
oped. Two exploratory experiments, or pretests, comprised launching a coffee 
slogan in a town and imitating a badge wearer in a boys’ camp. Since the 
activity rate waned harmonically in both cases, the harmonic logistic fit best 
in both the town and the camp as expected by the hypothesis. 


I. The Need for Waning Interaction M. odelst 


The Washington Public Opinion Laboratory is studying the principles 
of human interaction in the form of diffusing messages from person to person, 
using questions such as: How fast will a message spread (under specified 
conditions)? How far? How fully and how faultlessly? How effectively in 
arousing belief, retelling, and compliance? What conditions will maximize 
social diffusion? The principles should be stated in operational rules such 
as mathematical curves or other models. Each model should specify (a) the 
variables, (b) the social preconditions expressed as mathematical assumptions, 
(c) the consequent curve or formula which expresses their expected joint 
functioning, and (d) the procedures for testing the fit of the model. All 
this is a case of applying dimensional methods of analysis in the field of 


Social physics (1, 2). 
This paper deals only with the qu 
*A i he Committee on Social Physics of the 
paper read before Section K sponsored by the у: 
AAAS Conference in Boston, December 30, 1953. This research was supported in part by 
he United States Air Force under Contract AF 33(038)-27522, monitored by the Human 
esources Research Institute (now, Officer Education Research Laboratory, Air Force 
ersonnel and Training Research Center), Air Research and Development Command, 
axwell Air Force Base, Alabama. Permission is granted for reproduction, translation, 
Publication and distribution in part and in whole by or for the United States Government. 
iFor the Air Force, the project seeks to improve the leaflet weapon 1n psychological 
Warfare. The Air Force needs to know how to maximize the desired effects of the leaflets 
i i my or captive populations or on 
tion of this project and some of 
esis and dimensional theory, а speci: 


estions: How fast will a message 
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spread? What will be its growth curve relating Lene i ap d qe 
to time elapsed when all other factors are constant? It is limite cie: 
cases where an all-or-none act spreads through a homogeneous dace 
whose acting is either (a) steady in time or (b) waning since the sti ша 
that started it. Pretests indicate that the activity of retelling the Y^ p 
tends to wane inversely with time. Since these observations seem to т 
for either linear logistic curves or harmonically waning logistic curves, 
paper is a report on developing and testing such models. 


II. The Social Preconditions 


The social preconditions that are assumed in the simplest case may s 
stated in terms of three factors (at their first powers): (1) a human population, 
P; (2) an activity, A; (3) a period of time, T. ы | E 

; The relation at issue is that of the population-showing-the-activity 


to time (i.e., РА? о T). How does the number of knowers of the message grow 
with the time elapsed since the message started? 


The preconditions should also be stated in terms of these three factors 
only. This proves possible by stating them in terms of maximizing & ji 
portion of the zero-th statistical moment and minimizing second te 
or powers of these factors. The preconditions assumed may be loosely state 


here (and operationally further defined with testable indices below) as 
follows: 


1. The population interacts rando: 
interact—a meeting being det. 
influences. It seems likely tha 
population that is sufficiently 
ciples to work out smoothly. 


2. The time is sufficiently long to observe most of the growth or diffusing of 
the activity, i.e., from two hours up to three days. 

3. The activity is a novel all-or-none act of an 
The first oceurrence of a one-way all-or- 
simplicity in Studying interaction, here the retelling of a Message. 

4. The activity rate or “potency rate,” defined as acts-per-actor-per-period, 
is either: (a) steady fr 


Ога period to period—the linear case; or (b) waning 
with the time elapsed since the start—the harmonie case, 


mly. Everyone has an equal opportunity to 
ermined by many, small, and uncorrelated 
t this condition may be approximated in а 
large and homogeneous for probability prin- 


y person upon any other person. 
none activity is chosen at first for 


The acting may be expected to be st. 


eady when the stimulation is steadY 
for everyone, as in the case of some da 


or interests in each of the n succeed- 
hen tend to be reduced by a factor 
blic’s responding to such a punctifor™ 
with the time since it happened. 


ing time periods, the first event will t 
of 1/n in the public's attention. The pu 
stimulus will then decrease inversely 


ызы 
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The acting may also be expected to wane with time as a drop-out effect 
whenever there are individual differences in the speed and/or output of 
activity. If the output is fairly constant, faster actors will finish and drop 
out, leaving fewer and slower actors in the latter periods. Similarly, if output 
varies but speed is fairly constant from person to person, then those of low 
output will finish early and drop out, leaving progessively fewer actors. If 
both output and speed vary, the dropping out is accelerated and the average 
activity rate must wane as time goes on, the form of this curve of waning 
activity depending on the distribution of the individual differences. 

The interacting may wane with time due to the tellers getting more and 
more rebuffs. As more and more people hear the message, they will forestall 
its teller. He will stop trying to tell the message if he gets rebuffed often 
enough. This explanation, like that above, will result in a slackened pace of 
telling and of individuals ceasing to tell or dropping out. But the cause in 
rebuffing is social, whereas the cause in the other case is more specific to the 
individual. 

The two major conditions of human interaction here (namely, that the 
Population meet randomly and that the activity rate be specified), are 
highly general. They transcend any local culture or transient situation and 
apply to any all-or-none novel behavior. Personal, situational, and cultural 
Conditions may affect the numerical size of the activity rate of a particular 
act (or message) in a particular population and a particular situation. But 
if the given activity rate be either steady or waning in a large homogeneous 
Population, then a logistic curve of growth is hypothesized to be the necessary 
Consequence. 


III. The Mathematical Derivation of Models Matching the Preconditions 


A. The Linear Logistic. The derivation of the linear logistic curve 
assumes a large population (at least over 100 and preferably over 1,000) 
which is divided into two proportions: p, the proportion of knowers, and g, 
the proportion of nonknowers, at any moment, so that p + q = 1. Let them 


mix thoroughly during a unit period. This social interacting їз mathematically 


represented by an overtelling proportion, p, of knowers meeting knowers; 


a first-telling proportion, pq, of knowers meeting nonknowers, and sea 
Proportion, q^, of nonknowers meeting nonknowers. Assuming independen 


Probabilities, pq is the probability of a meeting in which the message can 


be spread. Let k represent the conditional probability of an actual telling. 


hus, pq is the probability of a knower and nonknower meeting, and k is the 


‘ ЖЕ » 
Probability of telling-if-met. k is observable from the "activity E P. 
“potency rate" defined as the hearers-per-teller-per-period. The pro m 
kpq, is then the net probability of a first telling during a unit period. 4 
is the expected rate of growth of the message- Written as the differentia 
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equation for an increment of diffusion in an increment of time, 


dp _ р 1) 
2i kpq. ( 

k is assumed to be constant from period to period as p increases and 4 
decreases. Integrating (1) over time gives the linear logistic, a symmetric 
S-shaped growth curve, 


T 
hoi. ciet Be Bote (2) 
1 + 2," 9: qo 
Do 


where po and qo denote the knowers and nonknowers at the start, and where 
k/4 is the slope at mid-date and mid-diffusion where the slope is maximal. 
Thus k shows the general steepness of the curve or speed of diffusion in а 
general way. 

"This simple logistic may be generalized in many ways, one of which is 
to substitute a function of time, f(), for the constant k and rewrite (1) 28 


d 
dr = fpg. 3 


This function is k;/L in the harmonic logistic equation below. The cumulative 
"augmented" logistic growth curve then is 


p= : 6 
1+2 ep | лаа 


A quadratic exponent giving a cubic logistic fits some of our data better 
than а linear logistie, but it requires four parameters instead of two, an 
parameters with no social interpretation at present. Still further generalizing, 
р and q may be replaced by integratable functions 


(4) 


d 
ae = fil) f(y) faq). e 


The subscripts here denote different functions. The dimensional family of 

these functions uses integral exponents, positive or negative or лего, a 

specify particular functions which describe many important social situations. 
The linear logistic may be written in discrete form as 


Dia = р, + kp.q, А (0) 

where p, is the cumulated proportion of knowers at time f. 
and £ + 1 the next unit period. 

This may be rewritten in terms of each successive. proportion's bein£ 

equal to the mean plus the (weighted) variance of the attribute in the рге 


, qı the nonknowers; 
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ceding period, i.e., 
Min = М, + ЕУ, . (7) 


For а quick trial-fitting by plotting the data on semi-logarithmic paper, 
the rectified logistic is convenient: 


In p/q = kt + In po/do - (8) 


If such a plot is linear, the slope, К, is the “potency” parameter. 
The logistic curve is factorable, since it is a simple product of the waxing 
exponential growth curve, 


pi = ре“, (9) 
times the waning exponential growth curve, 
1/4 = Woe (014 = qe **')- (10) 


The right-hand side of equation (3) is their product when k = kı + ka (see 
Figure 1). 

. The harmonic logistic is similarly factorable into the two harmonic 
exponential curves. All of these are factorable in both their differential equa- 
tion and their integrated forms. А 

A special variant form of the logistic becomes the Gompertz or “simplex” 
growth curve. The cumulated discrete linear logistic when k = 1 is also a 
special case of the Gompertz curve with а growth rate of 2. Thus (6) can 
be rearranged as 


i ga m P eee 
or 


A Fe 
dum c 3B Е 


Starting with до and squaring in each of the ¢ successive periods gives 


q= d- (11) 


This describes the decreasing of nonknowers, since g 1s less than unity 
and the exponent is greater than unity. If there are 99 per cent Aom е, 
at the start (qo), it takes some three unit periods to shrink them to less than 
one per cent, i.e., (.99) ^ < .01. ; 

в The кы Logistic. Next, if the activity rate k is not goss 
but is observed to reduce toward zero in time, the simplest form of descriptive 
curve is a harmonic series or inverse integers from one on up (or a hyperbola 
for continuous data). A straight line is simpler but cuts the axis and pes 
Meaningless when negative. The harmonic curve 18 preferable, ane i es 
away asymptotically. Its cumulative form is in terms of natura b ПГ; 
It has many social applications. One such is Zipf's size-rank rule an 
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hypothesized least effort principle. In cumulative form, it can often be 
interpreted as an extension to new fields of the Weber-Fechner relation. 
Such theoretical considerations, together with the fact that a harmonic 
curve has fitted the time series of activity rates closely in several pretests, 
determined its exploration here. 

. The harmonically waning activity rate in the logistic equation for 
interacting is specified by 


Ф — kupg/( + D, (12) 
where ko is the potency rate at time 0 and ko/(t + 1) is the potency rate at 
any time ¢. The ¢ + 1 means choosing + 1 as the time origin to avoid having 
the growth rate become infinite at t = 0, as it would in the simpler form, 
dp/dt = kpq/t. (With the simpler form, one can use the convention that 
growth starts at £ = 1, not at і = 0, to avoid an infinite growth rate.) This 


integrates to the cumulative growth curve, 


= «Ва Pr». (13) 
bteE qe v7 
0 


Di 


Here p, denotes the cumulative proportion of knowers at time ¢ and the 


Zero subscript denotes the starting moment (see Figure 1). . 

C. Units, Range, and Inflection Points. In order to standardize the 
abscissa, units of time in the asymmetric harmonie logistic curve and so make 
different growth curves more comparable, the first “half life" may be taken 
as a standard unit. This “chron,” as it may be called, is the time from no 
diffusion to half diffusion, from zero per cent of knowers, p, to 50 per cent 
of knowers. (The quarter life or any other fractional life might be used 
instead but with loss of simplicity.) The first “half life” does not equal 
the second. Since this curve is asymptotic at the upper limit only, the chron, 
or the period for the first 50 per cent to become knowers, 1s definite, while 
the remaining period is indefinite. In terms of chrons (with the origin at the 


absolute zero of diffusion) (13) becomes 
B LL ih. (14) 


q 
The chron may be viewed as an inverse function of the potency of 
message (when comparing populations of the same size), for it isa = 
of an activity rate. The activity rate states “aets-per-period;” the с п 
States а period-per-half-the-acts," or а time for 50 per cent of us popu Be 
tion to be diffused. The longer the chron, therefore, the more “impotent 


is the message. Thus, the chron or first half life becomes another ее и 
measure of strength of a message in a given population and situation. Du 
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The reasons for using the chron unit аге simplicity of the formula and 
flexibility of fitting. In chron units the formula drops the explicit po/do 
term and can be written in simplest form as 2/9 = Ё, oras р = qt’. In chron 
units the node where all the family of curves intersect (Figure 1) is at the 
half diffusion point where p = q = 5, This gives a wide range of shapes from 


to the data may be more attainable. 


o = 0 at to) and goes to 100 per cent шо 
For some purposes it may be truncated at 10 chrons, giving а standard tim 


Tange of 100 decichrons along the abscissa to match the population range of 


100 percentage points along the ordinate, Figure 1 shows these values of P 
at 10 chrons for various potencies. 


chron units when it has grown up to the sta Н 
Pa (on the curve fixed by one /). 


A difficulty in fitting these qt curves is to determine the absolute pier 
point, f , in time, which occurs when p, the growth, is zero, p, . But I 


th is when the starters, pa : 


тш 


STUART CARTER DODD 199 


may be taken as a first approximation for computing the chron and then К 
from the slope of the points. Using this best-fit estimate of Ё and the starters' 
proportion, pa , as p, one can then solve for і, getting a second approximation. 
Further approximations may be made by repeating these steps until the 
correction yielded is negligible. It seems likely that if the starters are few 
(such as less than one per cent), the error in the chron will usually be of 
the order of one per cent for values of k near unity. Then a single correction 
will be sufficient. It will always put the starters’ date at some positive fraction 
of a chron after the estimated absolute zero date and thus will lengthen the 
chron and flatten the growth curve slightly. 

The upper limit of the growth curve always presents à problem. Whether 
chrons or other time units are used, in the case of either the linear or the 
harmonie logistic, it is necessary to know P, , the terminal population, in 
absolute numbers. The difficulty is that there may be many different interpre- 
tations of P, such as: 


1. The census population in a diffusion area (which may be too large, 
since it may include undiffusable elements such as babies who cannot 
talk). 

2. The relevant population in a diffusion area which is thought a priori 
to be diffusable but which may have an undiffusable fraction cut off 
from each other by unknown barriers of physical, physiological, 
psychological, or cultural origin. 

3. The final diffused population, which may be unknown and have to be 
estimated as generally less than the relevant population. If the 
latter (such as “all adults") is known, it logically should yield better 
predictions than the larger census population. But insofar as the 
diffused population differs from the researcher's a priori judgment 
of the relevant population, the fits will be loose and the predictions 

poor. 


We have discussed two harmonic logistic curves, namely, 
de 2 engi teed (15) 
Where { is expressed in some conventional time unit, and 
(16) 


Ф = kpg/t, 
units. In the case of (15), we find 
dependence upon k: For po < $ 
for 


Where £ is expressed in chron or half life 
that the shape of the curve has the following 
the curve is concave downward throughout; 


1 
0<kS7— 2р 
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and for 


1 


ESI 


the curve is sigmoid with an inflection point at 


S do k = 1)” (17) 
r= (eka 


In the case of (16), where time is expressed in chron units, the curve is d 
downward throughout for i S 1 and is sigmoid for k > 1, with a point О 
inflection at 


f= (% = iE (18) 
k+1/ -* 
The harmonic logistic family of curves thus has a wide range of shapes, 
determined by k. 


D. “Timeless” Forms of the Harmonic Eq 


( 


Curves In general Fork = 1 
Harmonie logistic T m Kop 0/99, ал (19) Di =¢ (19a) 
Harmonic waxing dp = Тума 1 
exponential di = Fp MED (20) A ~ 2 dee 
Harmonie waning dp каљ (1) V: 
exponential dt r wm (1) (21) Ф =2 (218) 


{ез interaction, Le, a group 
nd (21) is one. Dimensionally 
› а plurel phenomenon. А 

The empirical or Social interpretation of (19a), (20a), and (21a) in 
the special case when k = | has not yet been fully explored. It appears 
that (20a) reflects constant growth, because the curvature of the waxing 


| 
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exponential is here exactly counterbalanced by the opposite curvature of the 
harmonie activity rate. In (21a); the harmonically waning exponential 
curiously has just twice the growth rate of the harmonie logistic in (19a). 
One possible social interpretation of the logistic (192) having d as its growth 
rate seems at present to be that the growth rate depends on two factors, 
one physical and one psychological, which here both happen to be numerically 
the same, namely, g. Thus, the growth rate shrinks as q shrinks when non- 
knowers become fewer and physically harder to find. But, in addition, a 
psychological factor may operate in that as the tellers get more and more 
rebuffs from the growing proportion of knowers, the tellers slacken their 
telling activity and this slackening keeps pace with 9, the proportion of 
current nonknowers. This explanation and alternative ones must be em- 
pirically tested. 


IV. Some Experimental Testing 


In order to begin testing the foregoing theory and to develop more 
definite tests, a dozen preliminary experiments were made in the first year 
of Project Revere. Two of these will be reviewed here as tests of the harmonic 
logistic hypothesis. [Some of our tests of the linear logistic have been pub- 
lished elsewhere (7, 8, 9)]. Both were designed to test linear logistic models 
in clock time units. In the first, however, the activity rate, k, was found a 
Posteriori to wane harmonically with removes, and its growth from remove 
to remove should therefore fit the harmonic logistic more closely. In the 
Second set of data to be reported, the activity rates were also observed to 
Wane, and therefore the harmonic logistic should again give a posteriori a 
better fit than the linear logistic. 

Time measured in removes or generations 0 
and is more free of diurnal and other rhythms w 
curves in clock or cardinal time units. Ordinal units seem apt to yield smoother 


curves which fit models more readily. But curves in cardinal time units are 
needed wherever practical prediction of growth in clock time is wanted. 
In both cases the reward offered as stimulus was expected to evoke steady 
acting throughout the whole period. Instead it produced a spurt of activity 
Which waned steadily—perhaps because of the overlaying or drop-out, or 
rebuffing mechanisms noted above. 
A. Coffee Slogan Diffusing in 
of every six in a village of 950 inha 
an interviewer ringing her doorbell on 
to retell it to their friends. A free poun 
housewife in town who might know the sl 
be canvassed later. 
son to person till 88 per cent of the house- 


_ This message spread from per pd h 
Wives knew it on Wednesday's census of households, determining the pro- 


f hearers is in ordinal units 
hich “overlay” the growth 


а Town. One randomly chosen housewife 
bitants was told а new coffee slogan by 
Monday morning. All were invited 
d of coffee was promised for every 
ogan when every household would 
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portion of knowers, the ordinates of the curve, or P values in Table 1, e 
3. In this census questions of who told whom, when, and where identifies 
the remove of each respondent and so measured the increment of new ew. 
at each remove of retelling and the potency of hearers-per-teller of each 
remove. The potency was found to wane with successive removes in ап 
harmonic curve. Thus, the harmonic subcase of the logistic should fit these 
data better than the linear case. 


TABLE 1 
Data for Testing the Harmonic Logistic Growth Model in "C-ville" 


= = аашаа MU C LUND Model ае АНН 


Removes Chron Observed Observed Expected increnent 
(ordinal time cumulated increment c t 
time population of of knowers d 
units) message knowers n 
Cm to Р АР А?! 
0 et ҮР. „ды r 
0 -84708 m 22.832 21.16% 
d 1.05876 111 31.50 39.52 
2 1.27ОШ 16} 28.80 25.68 
3 1.18212 178 7.61 8.93 
m 1.69380 180 1.09 2.81 
5 1.90518 18h 2.17 0.96 
Totals 1.9 "first 184 100.00% 99.36% 
half-life" housewives 


units 


III ceat E o: su 


1 remove = ,21168 chrons 


Loc Rd A E i л. E- 


The observed potency rates for the Successive removes showed a close 
ness of fit correlation coefficient; of -99 with the best fitting simple harmonie 
curve (a = k/t). (Sée Table 1). 

The correlation coefficient. of the increments of the observed growth gi 


message knowers with the increments in growth expected by the harmonie 
logistic curve (p/q = t) was also .99 CU MR .99). By the z test this 
7 is significantly different, from zero and also from our arbitrary standard 0 
close fit, namely, r = .9 at the 5 per с 

the z test is here, however, i 
message," 


а hypothesis was that (a) the closeness 
9 between the observed and the mode 


= 


model and data, was both descriptively small 
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expected data in uncumulated form, and (b) this 7 should be significantly 
different from zero at the five per cent confidence level. (This generally 
entails an r in cumulated data, such as is usually reported, above .99, but 
this test is insensitive and partly spurious since cumulating compels some 
correlation even in random series.) 

К The closeness of fit correlation of the uncumulated data to the linear 
logistic curve which is based on a steady activity rate was .37. This linear 
logistic hypothesis then was rejected. But the similar closeness of fit corre- 
lation (by a successive approximations technique in fitting) of the un- 
cumulated data to the harmonic logistic, based on a waning activity rate, 
was .99. Therefore the harmonic logistic hypothesis could not be rejected. 

An excellent fit of one model to given data does not preclude fits as 
good or better by other models. In fact, the waning random net variant of 
the logistic, which has a constant “potency ever," fitted these uncumulated 
data with r = .996 as reported by Professor Anatol Rapoport, who developed 
this random net model. Also, the logistic in discrete form (6) often seems 
to fit our data better than the continuous curve. 

B. Contagious Behavior in a Boys’ Camp. Three boys in a summer camp 
of 42 boys were given large yellow buttons with the name of the camp and a 
question mark in black lettering on them. These “starters” circulated 
during the noon rest period and, when asked, said that they had been told a 
few more such buttons were obtainable at the lodge where they received 
their buttons. The exact time at which each boy came in to ask for a similar 


button was recorded and so the growth curve could be plotted. The growth 


Tate of hearers-per-teller waned in the successive 15-minute periods during 
the two hours in which 39 of the boys came in for buttons. The total growth 
data were fitted to both linear and harmonic logistic curves (Figure 2). The 
closeness of fit correlations were computed on uncumulated data and were 
significantly different from zero at the one per cent confidence level. The 
correlation was .88 for the linear logistic and .93 for the harmonic logistic. 
The slightly better fit of the harmonic is in line with the theoretical ex- 
Pectations but the difference was too small and the underlying population also 
too small to warrant much dependence upon these findings. 

For both of the tests described here, the chi square test showed that 
the discrepancies between model and data were not significant at the five 


Per cent level: 


First Pretest Second Pretest 
Chi square | 6.163 Eu 
Degrees of freedom 4 Ж 
Probability .90 <p < .95 10<p<.- 


We conclude that in both pretests the looseness of fit, i.e., discrepancy of 
and statistically not significant 


amount of each parameter of that curve. 
wholly new situations. 


p=ACCUMULATED PROPORTION OF KNOWERS 
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Har monic Logistic Model 


o: 
Pep ale 


0 or in rectified form 


| In PA= 3.12 Int 


EXPECTED 


10 


ACCUMULATED 


T 1 CHRON GROWTH CURVE 

Time Observed Expected 

.00 .081 .000 

50 .33 .081 .030 

.65 .243 .207 

.98 .486 .485 

РА 1.3] ‚103 699 

OBSERVED 1.63 .838 .821 

30 1.96 919 889 

2.29 ‚973 .930 

B 2.61 1.000 .952 


The closeness of fit correlation 
Was .93 їп uncumulated data with 
the finally diffused population 
of 37 boys as upper asymptote. 


98 1.31 163 1.96 ~229 261 
t=TIME IN CHRON UNITS 


65 


... Diffusion of a Message in a Boys Camp Population _ 
Fitted by a Harmonic Logistic Growth Curve in Chron Units 


FIGURE 2 
at the five per cent level, The discrepancies here are unimportant practically 
and may be due to sampling error, 
The tests above were concerned with “hypotheses of form," not "hypo" 


theses of amount." These hypotheses asserted the form of relation betwee? 
the variables as being a linear logistic curve, ete., and did not assert the 


—which could not be expected i? 
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the variable discrepancy to be measured by the correlation. For this reason, 
the Pearson r is here almost identical with the intraclass т. The latter r is 
the more exacting descriptive statistic of closeness of fit, since it can approach 
unity only if the mean, variance, and rank order of the data agree with 
these moments in the model. 


m 
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A METHOD FOR OBTAINING AN ORDERED METRIC SCALE* 


SIDNEY SIEGEL 


PENNSYLVANIA STATE UNIVERSITY 


. A method is presented for collecting data which will yield a scale on 
which the entities are ranked in preference (ordinality), the distances between 
the entities on the scale are ranked (ordered metric), and all combinations of 


the distances are ranked (higher-ordered metric). "The sources drawn upon 
are von Neumann and Morgenstern (9), and lattice theory. An empirical 


example is given in which a higher-ordered metric scale is derived. 


. lf an individual is both consistent and transitive in his preferences 
With respect to a group of discriminable entities, it is possible by the method 
of paired comparisons to rank these entities in the sense of an ordinal scale 
of utility, e.g, A> B>0> > N (read: А is preferred to B, etc.). An 
Individual is consistent if he prefers the same entity of a pair whenever that 
Pairwise comparison is presented to him. His preferences are transitive if 
when A > Band B > C, then A > C. The entities involved (4, B, C, ete.) 
may be objects or actions. The utility of an entity is, roughly, the subjective 


value of that entity. 

Such a scale, however, gives no inform: 
the differences in utility between the enti 
others, has shown that knowledge of the magnitudes of these differences would 
Strengthen the measurement of the psychological attribute involved and, 
therefore, increase the amount of information obtained from the responses 
made by the individual to the stimuli presented to him. 


Coombs (2) suggests the label ordered metric for those scales which give 
not only an ordering of entities but also at least a partial ordering of the 
Coombs also presents a method, 


distances between the various: entities. : 
Which he calls the unfolding technique, for obtaining an ordered metric scale 
(J scale) from a rank order preference scale (ordinal I scale). ) 
A method is developed in this paper for obtaining ап ordered metric 
Scale of preference. This method is particularly suitable for the measurement 
of utility, which is a central concept in decision theory. The ordered metric 


ation about the relative sizes of 
ties. Coombs (2, 3, 4), among 
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scale derived by this method yields the following information: 


a. It orders the entities involved ie, 4 > B» (C » D x ees 
b. It orders the distances between the entities, i.e., say DE > AB > 
GD ora , where AB implies the difference in utility between A and B. 
c. It orders all possible combinations of contiguous distances between 
entities, i.e., say AB + ВС > BC + CD + DE, or AB > CD + DE. 

This method is not restricted by the number of entities to be scaled. 

Because this method yields more than a partial ordering of the distances 
between the entities (see b and с above), it is suggested that this type of 
scaling be termed higher-ordered metric. Coombs (4) seems to mean this type 
of scaling by his term ordered ordered. However, a scale may be ordered 


Sources of the M. ethod 


Von Neumann and Morgenstern (9, p. 17) have suggested that measure- 
ы of a person's utility in a Stronger sense than ordinality could be obtained 
1 

4. the person can always say whether he prefers one entity to another, 
and 
b. the person can also completely order probability-combinations of 
the entities, i.e., combinations of entities with stated probabilities 


A; 1/2), 16, 


› 


" ep ; 1/2), since A > B > C. In 
probability-combinationg of entities, вау (x, y; 1 /2), the prospect is of getting 
either x or y. The Probability of getting x is .50; the probability of getting Y 
is the remaining .50, The two alternatives in the probability-combination 

i the individual is absolutely certain of getting either 
ion. 


him to prefer getting A for sure to a 50-50 combination of В and С. 
Now suppose that the individual must choose between (B, B; 1/2) and 
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(4, C; 1/2). That is, he must choose between getting B for sure or getting 
a 50-50 chance at A or C. By making this choice, he yields new information. 
Tf he chooses the combination which gives him B for sure, his choice indicates 
that B is closer to A than it is to C. If he chooses (А, С; 1/2), then B must 
be closer to C than it is to A. This is fundamentally new information because 
the statement A > B > C told us nothing about the distances (differences) 
between the entities on the utility scale. Thus, the von Neumann and 
Morgenstern suggestions imply the possibility of measurement of utility 
on at least an ordered metric scale. 
The second source drawn upon in higher-ordered metric measurement is 
lattice theory. This source is not so centrally important as the first, but it 
does offer a heuristic device for indicating the minimum information necessary 
for achieving higher-ordered metric scaling. Birkhoff (1, p. 6, pp. 66-72) 
Suggests various diagrams which give a descriptive ordering of entities. 
Coombs (4, p. 4; 5, p. 475) suggests such diagrams and puts them to use. 
The lattice used here (Figure 1) not only gives a descriptive ordering 
of probability-combinations of entities (based on the individual's preference 
rankings) but also makes apparent which probability-combinations are not 
orderable from just a knowledge of the preference rankings. Süch probability- 
combinations will be called non-orderable. 
If an individual's preferences among five entities are A > B > 6 = 215 
E › the probability lattice is shown in Figure 1; where there is à connecting 
line between two probability-combinations, the higher probability-combi- 
nation is preferred to the lower (5, p. 475). In other words, if it is true that 
A>B>C>D> E, then any two probability-combinations on the lattice 
that can be connected with a line which is consistently going up (or down) 
can be ordered, with the higher probability-combination being preferred to 
the lower, e.g., 
(A, Е;®) > (B, Е; 9 
(А, D; ) > (4, Bit 
(B, D; 3) > (D, D; 2), ete. 

out the non-orderable relations, i.e., 


cannot be connected by a line always 
the horizontal-vertical dimension), 


^7 


Simple ranking tells us nothing ab 
any two probability-combinations which 
going in the same direction (with respect to 
е.к., 


(А, E; ? (B, D; 
(4, E; ) ? (В, С; 
(4, E; 3) ? (B, В; 2) 

(A, D;} ? (С, С; 2), etc. 


M ым 
— 
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It is the relations between these (non-orderable) pairs of probability-combi- 


nations which contain the information necessary to change an ordinal scale 
to a higher-ordered metric scale. 


(A, A; Ve) 


n 


(A, B; 2) 


М 
с 


(А, С; 1/2) (8, В; Ve) | 


X 
zí 
К 


(А, D; Ye) 


Y м, (B,C; v2) 
Ne v NN 


(В,Е; Ve) (С,05 2) 


Z 
PN 
га 


(C, Ej Ve) (D,D; '/2) 


x 
x 


(D,E; Ye) 


pa 


m 


3 Ve) 
Figure |, 4 


A careful perusal of the lattice in Figure 1, which is based on five entities, 
will disclose that there are 15 pairs of probability-combinations which cannot 
be connected by an always-rising or always-descending line. It can be show? 


that if N is the number of entities to be scaled, then ie d 1) umber 


gives the n 
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of non-orderable pairs of ili inatio 
ба, (5) ы m probability-combinations. In the case of five 
ife arid ene the non-orderable relations must be found in order to obtain 
sable fron E necessary for higher-ordered metrie scaling. In the present 
— ree to six of these relations (depending on the type of under- 
нь metric scale) are needed to achieve ordinal ordered metric 
(in M il en at most are required to achieve higher-ordered metric scaling 
opm combinations of the distances can be ordered). The fact that 
Min lia sat ү needed is important. The ordering of the remaining pairs 
Pion an icted after the minimum are used to obtain the necessary infor- ` 
Sn. ү аш ог failure of these predictions provides a check on 
Trek E hae metric scaling has in fact been achieved. 
will ben К ollows the general method of obtaining a higher-ordered metric 
the utlined. Then the operational definitions necessary for implementing 
general method will be given. Finally an example of the empirical use of 


the method will be given. 
General Requirements 


СЕ the p in the probability-combinations, e.g., (x, y; p) to be 
ios ve probability. [Ramsey (8) first suggested the importance of sub- 
Shack probability (degree of belief) in the measurement of utility. His 
E 14 resembles that given here in many respects and antedates von Neumann 
rent S UR EAR by more than a decade. However, the latter authors were 
TE 3 to make operationally clear а method for the measurement of utility. 
ae indebted to Professor Donald Davidson of Stanford University for 
monstrating the significance of Ramsey's work to me.] 
2. Find an event for which the person's subjective probability can be 


experimentally determined to be one-half. 
3. Require the person to rank, by the method of paired comparisons, 


the entities used. 

, 4. Require the person to state his preference between each non-orderable 
pair of probability-combinations. (As shown above, the majority of pairs of 
pe Pty cote may be ordered, as in the lattice in Figure 1, 
rom a knowledge of the person’s ranking of the entities. Step 4 is concerned 
with those pairs which cannot be ordered from this knowledge.) 

5. Observe those choices which will permit the determination of an 


ordered metric scale. 
hi 6. Observe those choices which will permit the determination of a 
igher-ordered metric scale. 
istent with the scale 


7. Check whether the remaining choices are cons 


derived from Step 6. If all of these choices аг 
Previously non-orderable (in 4 above) pairs of pro 


e consistent, then all of the 


bability-combinations are 
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consistent with each other and can be predicted from the higher-ordered 
metric scale derived. If Step 7 succeeds, then higher-ordered metric scaling 
has in fact been achieved. 


Operational Definitions 


Davidson, Siegel, and Suppes (6), in a study designed to measure en 
utility of money in the sense of an interval scale, developed an event whic h, 
for most people, has a subjective probability of one-half. Such an event is 
difficult to find because of the prejudices and superstitions which many 
people hold concerning familiar events, e.g., heads on coins, evens on dice, 
etc. The event used here is produced by means of specially-made dice. On 
three faces of the die, the nonsense syllable ZOJ is engraved, and on the 
other three faces ZEJ is engraved. Similar dice were made with pairs WUH, 
XEQ and QUG, QUJ on their faces. These Syllables were selected from 
Glaze (7), who reports these pairs to have practically zero association 
value. The dice were tested with each subject; in every case the expectation 
of zero association was upheld, i.e., each subject was indifferent about which 
nonsense syllable he would bet on or which one would be the winner. The 
use of these dice will be discussed further in the following section. . 

Careful and considered choices between the probability-combinations 
presented to a subject were assured by "realistic" conditions. That is, when 
books were used as entities to be scaled, the subject was assured of A». 
a book or books. The identity of the book he received was a function of al 
of his choice behavior. "Therefore he was highly involved in each choice. When 
amounts of money were used as the entities, the subject was given а sum 
of money (usually one dollar) at the start of the session. He gambled with 
that money, keeping all funds in his possession at the end of the session. : 

The essential device that, defines operationally how the subject's choices 
determine ordered metric scaling is a one-person game (6) in which the subject 


chooses between two alternatives, each of which is a probability-combination 
of two outcomes. The format for each offer is: 


Alternative 1 Alternative 2 
If event E occurs: you get w 


you get x 
If not-E occurs: you get z 


you get y 

; the outcome of event E determines the row- 

ch case not-E would be ZEJ. 

If the subject chooses alternative 1, then 
(w, 2; p) > (x, y; p). | 

If u(w) is read as “the utility of w” and is interpreted as the subjective value 

of w, i.e., its worth to the person, then (1) can be written 


p-uw) + (1 — p)-u(g > p-u(z) + Q — p)-u(y). dà 


The subject chooses the column 
Event E might be ZOJ , in whi 
Suppose w > z > y >z. 
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If p is understood to be subjective probability, and is known to be one-half, 
then (2) сап be written 


u(w) + u(z) > u(x) + uly); (3) 

and 
u(w) — u(x) > uy) — ue); (4) 

i.e., 
тт >, when (ш,2;р) > (f, y; p)- (5) 


That is, w and z differ in utility more than y and z. 

It should be noted that the distances are directed distances. That is, xiv is 
the negative of wz. To simplify comparisons, the convention has been adopted 
of always deriving the distance from the more preferred to the less preferred 
entity. [For example, from (3) we could get u(z) — uly) > u(x) — u(w). 
But since w > x > y > z, we multiply through by — 1 to get (4) as shown.] 


An Example of Higher-Ordered Metric Scaling 


A graduate student in psychology served as the subject. He was shown 


a collection of books and was asked to choose from them the five books which 
d to make this selection carefully, for 


he would most like to own. He was tol 
he would surely receive one of the books he chose at the conclusion of the 
Session. The books he chose, in the order of choice, were: (1) S. S. Stevens 


(Ed.), Handbook of Experimental Psychology, (2) E. G. Boring, A History of 
Experimental Psychology, (3) E- R. Hilgard, Theories of Learning, (4) E. R. 
Hilgard and D. G. Marquis, Conditioning and Learning, and (5) H. B. English, 
A Student's Dictionary of Psychological Terms, 4th edition. 

All possible pairs of these five books were presented to the subject 
orally, and he was asked to state his preference as each was presented. His 


choices were: 


Stevens > Boring (A > B) 
Stevens > Hilgard (А > D) 
Stevens > Hilgard 

and Marquis (А > C) 
Stevens > English (A > 2) 


Hilgard and Marquis > Hilgard (C > D) Hilgard 


Hilgard and Marquis > English (C > E) 
nd transitive; his choices would be 


Boring > Hilgard (B > D) 
Boring > Hilgard 

and Marquis (В > С) 
Boring > English (В > E) 


> English (D > Ё) 


The subject’s choices were consistent а 

ranked thus: A > B > C > D> E. 
Having stated his preferences among 

was introduced to the “game.” He was allowed to become 


the paired comparisons, the subject 
familiar with the 
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dice and the game by taking practice trials in which simple relations were 
offered, i.e., those relations which are connected by a line in the lattice [e.g-, 
(А, D; 1/2) or (B, E; 1/2)]. As a trial run he could choose between a 50-50 
chance of getting either Stevens (А) or Hilgard (D), or a 50-50 chance of 
getting either Boring (B) or English (E). 

The practice trials served not only to introduce the game but also to 
check on the consistency and transitivity of the lattice (Figure 1), because 
all of the choices on these simple relations should be predictable. The practice 
trials also served to check whether the subject’s behavior was consistent 
with a subjective probability of one-half toward the event. This was ascer- 
tained when the subject showed indifference as to which syllable would be 
the winner. That is, if the ZOJ-ZEJ die was used and the subject was willing 
to make his choice between alternatives 1 and 2 (without knowing or caring 
which of the nonsense syllables was to be associated with which of the 
outcomes) it was concluded that the choice was based only on the utility of 
the entities involved and was independent of the particular event giving rise 
to the outcome. 

, The subject was told to consider the alternatives, announce his choice, 
encircle that choice on a 3 X 5 card, and then turn that card face down. He 
was told that after all sets of alternatives were presented to him, the cards 
would be shuffled and he would then draw one card. The alternative which 
he had encircled on that card would be determined by a roll of the nonsense 
syllable die. Thus each selection made by the subject might be the crucial 
one, so each one had to be made carefully. 


(а) (4,C; 4) > (B, B;? 


2! 0%) (A, D;} > (B, C; D 
(0 8, D: > ©, 054) © (В, В; 3) > (4, 0; 
© (D, D;) > (C, E; y) (0 (60;9»(B,E;9 
@ (C, D;) > (B, Ej) ® 8,0 p» (4,3; D 
© (B, Dip» (4, E; p O (A, D; > (C, C; 8 


NIK ы 
E 
S 
è 


MD EOM ENS (APE; (D, D; 3) > (B, E; 9 
(0 (A, D; 4) > (B, В; з) (т) (C, D;3 > (A,E;9 
() (0,0; р > (4,5; D 


м — ae Te uO =а5 
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The inequalities (1) to (5) in the previous section show that these choices 
can be stated in terms of distances as: 


(a) AB » BC (à) AB » CD 
(b) BC > CD G) BE > AB 
(с) DE > CD (j) CE > BC 
(à) DE > BC (к) CE > AB 
(е) DE > AB (/ Аб > CD 
(f) DE > AD (m) DE > BD 
(g) AB > BD (n) DE > AC 

(0) CE > AC 


The first five relations—(a’) to (e’)—yield the ordered metric scale: DE > 
AB > BC > CD. When relations (/') and (g’) are also considered, we have 
necessary and sufficient information for a higher-ordered metric scale. 
Relations (h’) through (o) provide checks on the uniqueness of the higher- 
ordered metric scale derived by (a^) through (g^). 

The ordered metric scale given by (a^) through (e’) may be depicted as: 


А . В QUID E 


It is seen that the subject’s choices (stated in distances) in relations (№) 
through (/) may be predicted (i.e., checked) from a knowledge of just (a^) 
through (e^). However, in order to predict (m^), (n^), and (o^) the subject's 
choices in (f^) and (g^) must be known; knowledge of the latter provides а 
more powerful form of measurement. 
Inasmuch as all of the choices in 

predictable, i.e., were consistent with ch 
ordered metric scale derived from choices 
This higher-ordered metric scale may be depicted as: | 


A IDE XO D 


the relations (A^) through (o’) were 
oices in (a^) through (g^), the higher- 
(a^) through (g^) is unique and valid. 


E 


We now know not only that А > B>C>D>E (ordinal scale), and 
that DE > AB > BC > CD (ordered metric scale), but also that AE > 
BE > CE > DE > AD» AC» AB > BD > BC > CD (higher-ordered 
metric scale). 

Data on a utility scale of five entities (records or books or amounts 
of money) have been collected from 10 subjects. Of this number, nine have 


been consistent; therefore it was possible to derive a higher-ordered es 
scale for each of these nine. One subject showed inconsistencies 1n two o | е 
relations which “should” have been predictable; therefore a unique higher- 
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ordered metric scale could not be constructed for him. At present, the p 
‘of inconsistencies is being studied. One of the leads, suggested by be 
Radlow, is that inconsistencies are likely to occur in relations which vlr 
equal-appearing intervals or combinations of intervals. The findings on e 
inconsistent subject seem to Support this explanation. On the average, t : 
time required to obtain a person's higher-ordered metric scale for five entitie 
is twenty minutes. 
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BOOK REVIEWS 


Isapore Вілмех, Marvin Kocan, and PHILIP J. McCarray. The Industrial Mobility 
of Labor as a Probability Process. Ithaca: New York State School of Industrial and 
Labor Relations, Cornell University, 1955, xii + 163 pp. $3.00 paper, $4.00 cloth. 
(Cornell Studies in Industrial and Labor Relations, Vol. 6.) 


{ This is an investigation of how the theory of Markov chains can be used, or adapted, 
Or a description of the observed movements of the labor force within the United States. 
The phenomenon of labor movements has no direct connection with psychometries, but 
ш theory of Markov chains appears to a useful tool for the construction of models of 
RENDIR Psychometricians may therefore find interest in the book from a purely methodo- 
оріса] point of view. In fact, probably never before has so huge а statistical material been 
treated numerically by probabilistic models of the Markovian variety. The analysis is 
made expertly and described carefully to the last detail. The problems of handling the 
data, of statistical estimation, and of comparing theory with observations appear thus 
with great clarity. The shortcomings of the method are discussed with commendable 
frankness. 
" Consider, say, nine categories of employment а 
not covered by the preceding ones." Fix an arbitrary 
who at time ¢ are in category 7. At timet + 1 а fraction fii 
J (where fa + +++ + fio = 1). In a stable community t 


mately) independent of t. 

Denote by F; the ten by ten matrix with elements fii, and similarly by Fe, Ёз, °*" 
the analogous matrices for an observational period of length 2, 3, ++- . The simple Markov 
chain model assumes that the transitions from category ? to other categories constitute 
а random choice which is in no way affected by the past history (for example, of the time 
Spent in category i). If this were the case, the matrix Ез should be nearly equal to the 


matrix P of the theoretical transition probabilities, and F; , Fs, ::* should be close to the 


nd add the tenth category entitled 
time unit and consider the workers 
of them will be found in category 
hese frequencies will be (approxi- 


Powers P2, Рз, +++ of the matrix P. 

The authors show how to estimate P and find that in actual practice the diagonal 
elements of Ез, Fa, © consistently and significantly exceed the corresponding elements 
of Рз, P3, ... , This indicates that (contrary to the assumption underlying а Markov 


Process) a prolonged employment in a category decreases the probability of a move into 
another category. Accordingly the authors refine the model by assuming that the entire 


Population consists of two strata, the "stayers" and the *"mpvers." The stayers never 
move whereas the movers are subject to а random process of the type just described. If 
the relative sizes of the two strata are p andq = 1-р respectively, the present model 
Predicts that Fa = pI + gP”, where I is an identity matrix. However, this model actually 
Overestimates the diagonal elements of Fn . The present model is a special case of an exceed- 
ingly flexible and useful model to which the authors call attention. Instead of assuming 
that the stavers never move, we may assume that each of the movements in each stratum 
are subject to a Markov process as described above with matrices P and Q, respectively. 
Now we should have F, = РР" + 40". Instead of two strata one may consider a larger 


i i i indicated 
number of st ining higher accuracies. (Further modifications are 1n 
xis, Haa ut eful for learning theory even more than 


in the last chapter.) Models of this type could be us y 
for labor movements (where probably the after-effect of the past history 18 50 pronounced 
that higher-order Markov chains must be introduced). 

William Feller 


Princeton University 
New York: Wiley, 


1955. *Cf. R. R. Bush and F. Mosteller, 


Stochastic Models for Learning. 
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Paut E. Mzznr. Clinical and Statistical Prediction. Minneapolis, Minnesota: University of 
Minnesota Press, 1955, pp. x + 149. 


surprising, therefore, that the clinician and the statistician see things in different ways and 
have difficulty in communicating with one another. Mech] has attempted, in his present 


between these groups. 

Speaking as a psychometrician-statistician, I feel he has been quite successful. At 
least, there was little that I felt inclined to quarrel about in his presentation, and I feel I have 
а more sympathetic understanding of the clinician’s activities as a result of my reading. I 


have not obtained the reactions of any thorough-going clinicians to find whether they felt 
equally satisfied. 


There is not, as Meehl points out, one single clear issue as between the clinically 
oriented and the statistically oriented psychologist. Rather. 
lems. Thus, one issue concerns the valu 


, ir general trend seems to definitely favor mechanical 
to be predicted is some pre-established set of socially 
ories as level of academic grades, progress in recovery 


om any psychometric device, 


Teachers College, Columbia University Robert L. Thorndike 


— — n UAR TTE ЕЕЕ ee eee ee ee 


PSYCHOMETRIKA—VOL. 21, NO. 3 
SEPTEMBER, 1956 


THE RELATION BETWEEN INFORMATION AND 
VARIANCE ANALYSES* 


W. R. GARNER 


THE JOHNS HOPKINS UNIVERSITY 
AND 
WiLLiAM J. McGILL 


MASSACHUSETTS INSTITUTE OF TECHNOLOGY . 


Analysis of variance and uncertainty analysis are analogous techniques 
Íor partitioning variability. In both analyses negative interaction terms 
due to negative covariance terms that appear when non-orthogonal predictor 
variables are allowed may occur. Uncertainties can be estimated directly 
from variances if the form of distribution is assumed. The decision as to 
Which of the techniques to use depends partly on the properties of the cri- 
terion variable. Only uncertainty analysis may be used with a non-metric 
criterion. Since uncertainties are dimensionless (using no metric), however, 
uncertainty analysis has a generality which may make it useful even when 
variances can be computed. 


I. Introduction 


Shannon (5) has defined amount of information by the formula 


HQ) = — 1; »0 log: 29, o 


Where y has r discrete values, and p(k) is a probability distribution defined 
over y. In communication theory, y is considered a source of signals, and 
the measure H represents the average number of binary digits required to 
code or store one of the signals. A broader interpretation, however, makes 
H a parameter which measures the non-metric variability of any probability 
distribution. Н has a value of zero when the probability is concentrated in a 
single category and is maximum when the probability is uniformly distributed 


Over all categories. 

Psychologists have been attra 
measure and the obvious application to si 
be computed. Since this use of the measure i 
cal properties and not with its interpretation in 


cted by the non-metrie character of this 
tuations where variances cannot 
s concerned only with its statisti- 
communication theory, we 


1 *The work of the senior author was supp 
l, between the U.S. Office of Naval Research and The 
is Report No. 166-I-192, Project Designation No, 


219 Í 
| 


Gureat' Ednl. 
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i . We 
Shall use the more general term uncertainty, U, to refer to the measure 


: М can 
shall show that uncertainty has many of the Properties of variance and 
be partitioned into components as variance can. 


a criterion is predicted from. ue 
or more predictors. The development will be presented for the three-varia 


i iterion 
case, where the problem is to determine to What extent values of the criteri 
2 


assumption does not limit-any of the pri t 
In the three-dimensiona] Matrix, n;,, refers to the number of ane A 

a single cell; n, refers to t mber of cases having the ith va 

- refers to the tota] number of cases having 


f w. Similar subscripts indicate other combinations of e 
n with no Subscript indicates the total number of cases i 
the matrix. In а i 1 


j indi nd 
nalysis of Variance formulas, 9 indicates a mean value, A 
i S used for mean values of the su 


ПІ. The №, ature of Uncertainty Analysis 
Analysis of variance can be Considered as 
the variance of the criterion variable is partitio 


able. The раша T uic Onstrate the nature of 
uncertainty partitionin: 


1 it to Variance partitioning. This 
nd explained in Table 1, The results of uncertainty 
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partitioning specify sources and magnitudes of variabilities as well as amount 
of categorical discrimination available. These uses are explained more fully 
by Garner and Hake (1) and by McGill (3). А 
d The ль can be considered as sample estimates of p(t, 7, k); we can use 
lese sample estimates to test various hypotheses about the parent dis- 
tribution. For example, suppose we wish to test the hypothesis that both 
predictors are independent of the criterion, ie., 
pli, j, ® = pli, 209. (2) 
lt can be shown (3, 4) by using the likelihood ratio, that when hypo- 
thesis (2) is true, [1.3863 nU(y: w, x)] is distributed approximately as chi 
Square. Independent tests can be constructed in the same way for each of 
the predictors separately as well as for the interaction between predictors. 
The approximation to chi square is of the same order as the familiar chi- 
Square contingency test so that, in effect, uncertainty analysis is analysis of 
Contingency chi square. Miller and Madow (4) discuss this aspect of un- 
certainty analysis more thoroughly. 


IV. The Orthogonal Case 

e and in uncertainty analysis, the ex- 
ctions. Orthogonality is defined 
This requirement is met when 
can be predicted correctly from 
when 


, Usually in analysis of varianc 
Perimenter tries to set up orthogonal predi 
аз zero association between the predictors. 
the cell frequencies in the matrix of the т. 
the row and column marginal frequencies, i.e., 

Nie тылы, (8) 
n 


Uncertainty Analysis 
'The partitioning of U(y) in uncerta 
U(y) = Uy: w, 2) + Ue); 


inty analysis is illustrated by 


(4) 


Where the uncertainty measures have the definitions given in Table 1. The 
Second term on the right-hand side of (4) is the error uncertainty, i.e., the 
amount of uncertainty in the criterion y remaining after the predictable 
Uncertainty has been eliminated. The first term on the right-hand side of 
(4) is the predictable uncertainty; i partitioned into com- 
Ponents 


t in turn can be 


Uly: w, 2) = Uly: w) + U(y:z) + U(y: wa). (5) 
1. A feature of uncertainty analysis 


"These terms are also defined in Table П 
the uncertainty in y predictable from 


i the interaction term U (y: 02). This is 
Unique combinations of w and z. 


Equation (5) describes a process t m with the 


hat is identical in for 


TABLE 1 


Symbols, Formulas, and Definitioas Used ia Threc-variable Uncertainty Analysis 


The criterion variable, y, 


is assumed to be non-metric. The predictor var- 


iables, w and x, are categorized and may or may not be metric variables. 


Definition 


Total uncertainty: The amount of uncertainty in the criterion 
vadable, y, 


Conditional uncertainty: The amount of uncertainty in y when 
one predictor variable, w, is held constant, 


hj 
ij 


Error uncertainty: The amount of uncertainty remaining when 
both predictor variables, w and x, are held constant, 


Шу) - Uy) 


Contingent uncertainty: The uncertainty in y due to the predic- 
tor variable, ш. 


Uy) - Uu.) 


Partial contingent uncertainty: The uncertainty in y due to the 
predictor variable, x, when the predictor variable, w, is held 
constant. 


Uy) - Uy) 


Multiple contingent uncertainty: The uncertainty in y due to the 
joint influence of the predictor variables, w and x, 


Чу) + Uy) + Us) - Чу), 
or Ulya) — Uy:x) 


Interaction uncertainty: The uncertainty in y due to unique com- 
binations of the predictor variables, w and x, 


léga 


VAIULAIWOHOASA 
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Analysis of Variance 


8 is generally appropriate when the criterion variable 
e, Le., one allowing only nominal scale values (cf. 6). 
The predictor variables may be categorical, or they may be metric variables 
which are categorized for Purposes of analysis. If the criterion is ‘a true 
metric variable, i.e., one having at least, the Properties of an interval scale, 
We can compute. variances and perform analysis of variance. The predictor 
variables must be categorized in any simple form of the analysis of VADO 

Equations describing analysis of Variance are essentially identical to 
those of uncertainty analysis, The defining equations are given in Table 2; 
except for the fact, that variances are computed from squared deviations, 
whereas uncertainties are computed from log-probabilities, the equations 


are identical to those in Table 1. The Partition of the variance of the criterion 
can be written: 


Uncertainty analysi: 
y is a categorical variabl 


ide of the equation are the predictable 
and the error Components of the total i 
can be broken down аз before: 


nty analysis in every respect except that in 
ance the Criterion Varia 


analysis, 


a negative covariance term 
On effect, 


W. R. GARNER AND WILLIAM J. MCGILL 225 


Uncertainty Analysis 


| Tt is not difficult to show that the interaction uncertainty in (5) can be 
written 


U(y: we) = U,(w:x) — U(w: 2). (8) 


This form of the interaction term shows at once that interaction cannot 
be negative with orthogonal predictors since orthogonality requires that 
U(w: x) = 0. 

. In the non-orthogonal case, however, U(w: x) will be greater than zero. 
With certain combinations of cell frequencies, the contingent uncertainty 
between x and w can be larger than the partial contingent uncertainty— 
resulting in negative interaction. A simple illustration of this principle is 
provided when each value of w is paired uniquely with each value of т. Now 
U(w: x) is as large as it can be. Furthermore, U,(w: х) cannot be greater 
than U(w: x) since U(w: x) is the maximum contingent uncertainty that can 
be obtained from a contingency table involving w and z. Equation (8) shows 
that the interaction will never be greater than zero. Ап identical result is 
obtained in the variance analysis when the predictors are completely con- 


` founded. 


Analysis of Variance 

It is usually assumed that the components of the total variance in 
analysis of variance must be positive. This is true only in the orthogonal 
case; if an analysis of variance is carried out with a non-orthogonal ex- 
perimental design, using the equations given in Table 2, negative interaction 
terms can occur. 


_ To show how this happen: 
interaction variance for the general case. The equa 
/ 


s, we now analyze the components of the 
tion is 


V(y: wa) = i 2 naa. — 8e. — 9 + 9° Н 
2 nidis р), 
x 2 (s. EET Ja... Qg..-9 


It can be seen that the interaction variance is composed of two parts: 
the first part is essentially the interaction variance in the orthogonal. case; 
the second part is a negative covariance term. This term must be zero in the 
orthogonal сазе [see equation (3)], but in the non-orthogonal case it cannot 


be ignored. The redundancy introduced by non-orthogonality is illustrated 
clearly in multiple regression. No tted, but a correc- 


А interaction term is permi r 
tion for non-orthogonality mus the predictor 


t be introduced whenever 
variables are correlated (cf. 2). 
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VI. Effects of Non-Orthogonality 
Our discussion of non-ortho 


ysis of variance in which 
the predictors w and т are completely confounded, The two main-effect 
variances and the interaction will all be identica]. The covariance term in 


ut two of these variances, but we do 
not know which two of the varia 


Хас& interpretation of the 
component variances impossible, 

The multiple contin; 
can be computed direct] 


Y of events in 


i i arate 
Predictabilities obtai mg up the sep: 


its; ng events disp] 
units in the time 5 pla; 


: y be used to esti inties 
hat y is normally distributed, Маз 
Shannon (5) has Shown 

can be specified as 


E log, m, á 0) 


yed by one or more : 
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analysis of variance. For example, 


est Us.(y) = 3 logs 2neVue(y) — log, m (11) 


is А " Д € 
the error uncertainty estimated from error variance. From definition 


(6) in Table 1, and from equations (10) and (11), we can write 

est U(y: w, 2) = $ loge [700/700]. 2) 

i Thus, it is relatively simple to estimate the multiple contingent un- 

po from the appropriate variances. The expression on the right-hand 

Š: e of this equation is reminiscent of the multiple correlation ratio (v). 
We can, in fact, write 

est U(y: w, 1) = — 2 loge [1 — ly: w, ®)]. (12-А) 

а Estimated uncertainties have the properties of additivity observed in 

pea uncertainties. Consequently, the expression on the right-hand 

ide of (12) can be partitioned into three components, each of which is based 


on its equivalent variances as follows: 

est U(y: w) = $ 108 [V@)/V Кел 
2 log; [V(y)/V-WI], 
3 log. (IV «()* VVO) Ve (15) 
ut some of the differences between 


(18) 


est U(y: 2) = (14) 
est U(y: Wz) = 
These estimating equations point o 
Uncertainty and variance. If (15) is used to estimate the interaction Un- 
certainty when the interaction variance is zero, cases can be found in which 
the estimated interaction uncertainty (and the computed interaction un- 
certainty) will not be zero. Converse cases (1.е., Zero uncertainty interactions 
With finite variance interactions) can also be found. These apparent con- 
tradictions are due to the fact that variances and uncertainties, while anal- 
ogous, do not measure exactly the same characteristics of probability 
distributions. Uncertainty analysis depends on the number of categories 
occupied by a distribution. Variance analysis depends on the weights or values 
attached to these categories. 


VIII. Application of the Measures 


We have now shown that uncertainty analysis and analysis of variance 
are equivalent in many respects; the question naturally arises as to when 
One should be used in preference to the other. This decision depends on the 
Properties of the data and the assumptions the experimenter is willing to 
make. If the criterion variable y has only the properties of a nominal or 
Ordinal scale, then only uncertainty analysis is permissible. Uncertainty 
analysis has the greater generality and requires no assumptions about metric 


Properties of the criterion. 
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MEASUREMENT OF SUBJECTIVE VALUES 
HAROLD GULLIKSEN 


PRINCETON UNIVERSITY 
AND 
EDUCATIONAL TESTING SERVICE 
, Four different value laws are developed and tested by using them to 
predict the scale values of composite stimuli from the scale values of their 
components. These four laws are: an additive law, а square-root law, & 


logarithmic; and a negative exponential law- They are tried out on a set of 
ood preferences by means of Pearson's Method of False Position. The nega- 
the best fit to the data but 


tive exponential law of diminishing returns gave 
was not markedly better than any of the other laws. 


The purpose of this study is to show that laws relating subjective value 
to amount of commodity may be studied by an extension of the usual psycho- 
physical scaling methods. Four different value laws will be developed and 
tested with a set of experimental data on food preferences. 

f The psychophysical scaling procedures, such as paired comparisons, 
or example, may be used to distinguish between various types of laws 
expressing value increase as а function of increase in amount of the com- 
modity. These procedures are applicable even when no physical measurement , 
of the amount of the commodity can be made, and when the scaling procedure 
necessitates measuring from an arbitrary origin. 

Testing each of these laws of value increase also involves а corresponding 
determination of an origin, or point of zero value. Various psychophysical 
methods of determining a zero point have been presented in the literature. 
An additive law of value increase has been used for this purpose (cf. 1, 4, 
and 7). It will be shown here that other laws of value increase may also be 


Used to determine an origin or point of zero value. y 1 

The data necessary to test these formulations are obtained by using 
a preference schedule like that illustrated in Table 1. The subject 1s asked 
the usual paired comparison question, «Which do you prefer, ? OF 1?” How- 
ever, in addition to the single stimuli 4 and j composite stimuli of the form 
& and j) or (g and Л) are used. The subject is asked questions of the form, 

Do you prefer (i and j) or g?” as well as, “Do you prefer (i and j) or (g and 
h)?” (15). 

Do these different stimuli, designated by ?, 
ete., behave as if they are different amounts О 
Subjective value v is given by some function, say Y 
function, the experimental device of utilizing the compo 

229 


j, 9, h, Сапа 7), (g and h), 
f some commodity x whose 
= f(x)? For any given 
site stimuli of the 
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type (i and j) may be indicated by writing 
v; = f(z); v; = f(z); and v; = f(z; + х1). (A) 


For example, if the items 7, 7, and (¢ and 7) were bundles of dollars, such 
that the number of dollars in each bundle were known to the subject, but 
not to the experimenter, then the experimenter could raise the question 
asked here. Do these different bundles and their combinations behave as 
if they were different packages containing different numbers of dollars 3 
which were known to the subject and were related to the subjective value 
v by some function v = f(x)? 
It is of interest to note that since 
4 


ж = Sf"); T; = £70); and =; + T; = S's); (B) 


we have 


ПЕКО) = f^) 3k 70). (С) 
From the viewpoint of psycholo; 


gical scaling, we do not obtain v's, but 
instead obtain s’s which differ fro 


m the v’s by a constant, so we may write 
Iii — À = fs, — ©) + f(s; — с). (D) 


designated f^', such that an 
ale values as indicated then 


T's) = g^) + д7), 
we may regard it as defining another commodity amount y, 


of interest to ask about the relationship between x; and y; 
and f^. Substituting (A) in (Е), 


(Е) 


= g^ (v,). It is 
or between g^! 


Ie: +2) = g"f(z) + 9 f). 


(F) 
A theorem in functional equations states that if 
F(X + Y) = F(X) + F(Y), © 
then 
F(X) = aX, (Н) 
where a is a constant coefficient; This theorem was 


Hamel (6) and Sierpiński (11). Applying solution (H) 


9^ f(x) = а(х). 


eo x O a 
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Utilizing (B) gives 
g (v) = af u) or y = ат. 

In other words, the "different" value laws found in this way would differ 
only in the unit of measurement used for the commodity. It should be noted 
that this statement holds only if equations (B) and (E) hold without error 
for values of v; , v; , and v; . | | 

In order to test different value laws, the theoretical formulation will 
, need to show the relationship between the values v; and v; for the component 


stimuli and the value v,; of the composite stimulus. Let us see what different 
laws of value increase imply with respect to this relationship. 


Four Laws of Value Increase 
Logarithmic Law 


If it is assumed that the rate of change in value v is inversely pro- 
portional to the amount of the commodity т, we have the differential equation 


à dv/dx = k/x. (1) 
Integrating this equation and setting x = % when v = 0, we have 
v = k log (x/2»)- (2) 


This derivation of a logarithmic law of value increase was given by Thurstone 
(13). 


What does this logarithmic value law imply with respect to the relation- 
ship between the values of the component and the composite stimuli? In order 
to determine this, we note that 
k log [(;/zo) + (0/20), 

k log (x;/2o), (3) 


1 


Uii 


0; 


І 


апа 


| 


v; = k log (x;/x0)- 


Eliminating x; and х, among these equations enables us to find an expression 
for the composite value v;; in terms of the values v; and v; of the components. 
Thus, we find that 


gi grist д et. (4) 


'These equations do not require the measurement of amount of commodity 
x, but require only a set of value measurements v. However, the usual scaling 
data give neither the v’s nor the x’s but only scale values, s, Which differ 
from v by a constant. Let C be the scale value for which v equals zero. 
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By making the substitutions 
0; = 8; — C, 
v =s; — C, (5) 
апі 


9; = 84 — C, 


let us determine the type of relationship among the scale values of the 
component stimuli and the composite stimuli that is implied by the log- 
arithmic law of value increase. Substituting and simplifying gives 


eii/* = eti/* дь e'i, (6) 


This equation gives the interrelationships among the experimentally de- 
terminable scale values which are implied by the logarithmic law of value 
increase. 

There are some interesting and disconcerting things about this law. 
It contains the parameter k, which is involved in such a way that it cannot 
readily be solved for explicitly. Thus, it remains as an annoying trial param- 
eter in any attempts to verify this equation. It is also interesting to note 
that the equation does not contain С at all. In other words, for the logarithmic 
law of value increase the additive constant C cannot be determined. Any 
additive constant is consistent with the relationships among the component 
and composite scale values. Likewise, any value of те is consistent with these 
relationships, since for this law, 2, functions as a unit of measurement for т. 


Square-Root Law 


Tf it is assumed that the rate of change in value v is inversely proportional 
to the value level already attained, we may write the differential equation 


dv/dx = k/2v. (7) 
Integrating and setting ж = £o when v = 0 we have 
к= Vika — a). (8) 
This derivation of a square-root law of value increase was given by Thurstone 
(13). É 


'To determine the implications of this square-root law Wi о 

: t 7 ith respect to 

the relationship between the values of the сот опе and com’ ] 1 1 
j р posite stimuli, 


ММ = ху) 
= V k(t; — a) 


= 
Ш 


SS 
I 


(9) 


= 
І 


—— ———— фр 
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ns enables us to find an 


Eliminating х; and х; among these equatio 
the values v; and v; of the 


expression for the composite value 2;; in terms of 
components. Thus, we find that 


y =й to; Кш. (10) 


In order to determine the relationship among the scale values implied by 


this equation, we substitute (5) in (10), giving 
s; — si — sj = 2С( — 5 — si) + С? kn. (11) 


This equation gives the interrelationship among the experimentally deter- 
minable scale values which is implied by the square-root law of value increase. 
The plot of the quantity on the left side of the equation against the quantity 
in parentheses enables one to find C from the slope of the line, and also to 
find kx) by subtracting the square of half the slope from the intercept. 


Separate values for k and хо cannot be determined. 


Negative Exponential Law 


Tf it is assumed that the rate of change in value v is directly proportional 
to the difference between the value level already reached and an asymptotic 
value level A, we have the differential equation 


dv/dx = КА — 9). uo 
Integrating and setting v = Xo when v = 0 gives 
veru uaa (13) 


This is the familiar negative exponential law of diminishing returns used in 
economics (see, for example, 8 and 12). It has also been suggested by a 
number of writers as an equation of the learning curve (see illustrations 
cited in 3). If we apply it to the component stimuli “г” and “j” and also to the 
composite “i and j", we have 


tzo, — k: 
jM E Age, 


ely = Ade, (14) 
S a em Auge Er ЧЕ 
Eliminating 2, and 2, to find v;; as a function Oft: and v; gives 
(15) 


оне es VV AC d hee CO) Cm vj. 


If v, is plotted against v;; for a given value of v; , the result is à straight 
Jine. If this plot is made for each of the values of v; , the result is à family e 
straight lines. We also note that if either v; OT Vi is equal to 4, (nen the right" 
hand term of the equation vanishes, giving si = 7" Thus, the indicated plots 


constitute a pencil of straight lines intersecting in the point (А, A). Again 
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we may substitute for the v's in terms of the s's (equations 5), obtaining 


85 = A C — Q/Aye (A + C — sY(A БС s). (16) 


Again, if s; is plotted against s, ; for a given value of s; 


; ; the result is a straight 
line. If such a plot is made in turn for each of the possible values of 5; the 


result is a pencil of straight lines intersecting in the point (A+ C, A + C). 


If ж = 0 this series of straight lines may be used to give the values of А and 
ofC. ` 


Linear Law 


If we assume that the ra 


te of change of value is constant, we have the 
differential equation 


dv/dx — k. (17) 
Integrating and Setting x = z, when v = 0, we have 
v= K(x — ху). (18) 


If we solve for the intern 


elationships among the component and compos- 
ite scale values implied by t 


his equation, we find that 


985 = 8: + 8; — C + km. 
This relationship has been deri 
lished study (15). According ti 


The Food Preference Experiment 


Let us now consider the type of data that is used for these value studies, 
Food preferences were studied. 


Pairs of single items ar 


› tor j. Then pairs of what we shall 

term composite items are 
Beef and Steak уз. Tongu 
and composite items, such as Ste 
three typical items in the schedule, 
The set of 5 compone 


tween single 
ble 1 shows 
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TABLE 1 


Semple Questionnaire Items 


Г] Roast Rib of Prime Beef 
El Roast Loin of Pork 

Roast Rib of Prime Beef 

39. Sirloin Stesk 


Boiled Smoked Beef Tongue 
Loin Lamb Chop 


Sirloin Steak 


Roest Loin of Pork 
Boiled Smoked Beef Tongue 


incomplete data. A least squares procedure was used (5) for scaling à paired 
comparisons matrix of incomplete data. 

The scale values obtained range from .000 for tongue and pork, the 
least preferred item, on up through 1.043 for lamb, to 2.622 for the composite, 
beef and steak, which was the most preferred item. The complete set of 
scale values is shown in Table 2. 

The fact that these 15 scale values give a good fit (8) to the 55 paired 
comparisons judgments shows that persons can make consistent judgments 
about preferences for composite stimuli along with single stimuli of the type 
used here. Thus, it is experimentally feasible to present in a single schedule 
comparisons of the (i vs. j), (i vs. g and Л), and (i and j vs. g and h) types. 
Any set of concrete objects or even abstract concepts can be dealt with 
according to this pattern. 

In Table 2 we notice that the value of tongue and lamb is higher than 
tongue alone, pork and lamb is higher than pork alone, beef and lamb is 
higher than beef alone, and lamb and steak is higher than steak alone. Thus, 
the value of any item is increased by forming à composite with lamb. The 
same holds true for beef and steak. Thus, lamb, beef, and steak are all positive 
values. However, now look at pork. The value of pork and steak is lower 
than the value of steak alone, of pork and beef lower than beef alone, of 
pork and lamb lower than lamb alone, and the value of the composite tongue 
and pork is lower than that of either of the components. Thus, from the 
purely ordinal characteristics of the scale, it seems clear that pork has a 
negative value. The same is true to an even greater extent for tongue. The 
zero point is between pork and lamb since pork and tongue are negative 
and lamb, beef, and steak are positive. 
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TABLE 2 
pEood&Ecefexence esperimento 
Scale 
Stimuli Values 
Tongue and Pork +000 
Tongue 17 
Tongue and Lamb +270 
Pork -541 
Tongue and Beef -830 
Pork and Lamb -928 
Lamb 1.043 
Tongue and Steak 1.088 
Pork and Beef 1.148 
Beef 1.746 
Pork and Steak 1.780 
Lamb and Beef 1.993 
Steak 2.197 
Lamb and Steak 2.324 
Beef and Steak 2.622 
——————— Án 
Generalization to Т. nclude Positive and N egative Values 
For this particular set of data there is thus clear evidence that some 
of the component values are positive i 


yr і 
ij 


where v; = s; — Civ; = 8; — C. 


Оза э РЕ СРР рр 


? 
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This formulation merely says that v; and vj are added if v; and v; are 
of the same sign and subtracted if v; and v; are of unlike signs. 'The square 
root of the sum is then given the sign of the larger value. If v; and v; are both 
positive, (20) is equivalent to (10) or (11). If vi and о; are both negative, 
(20) is equivalent to a reflection into the quadrant where S: , 8; ; and Si 
are each negative. If v; and v; are of opposite sign, (20) gives an interpretation 
that is consistent with the previous cases. "" 

In order to state the logarithmic law for a series of either positive or 
negative values we have analogously with the square-root law: 


p es 177877 Vecl7k ViVi леик C (21) 
Si; late E e + EA TC, 
where 0; = s; — C;v; = s; — C. 


Again this formula merely states that e is taken to be a positive power 
for either positive or negative values. The resulting powers are added if 
v, and v; are of the same sign and subtracted if v; and v; have: different 
signs. The logarithm is then given the sign of the larger value. In contrast 
to (6), it is now possible to determine C for (21) since the combination. of 
negative and positive values is involved. 

The negative exponential law may be restated for the case in which both 
negative and positive values are involved. Let ж = 0; thus 


A-—v ci 


A» e (wherev — s — C). (22) 
'The formulas are easier to work with if we define B as the asymptote ex- 
pressed in terms of the s-scale, just as C is defined: 
B=A+C. 


Let us assume that the positive (upper) asymptote B* may vary independently 
of the negative (lower) asymptote B^. Then let us define 


В, B"—s1.. jt PILIS (23) 


a VB = Cal qu 
* + 
The superscript signs are used to indicate that the positive sap 
is used if s; > C Ge., for s^); the negative asymptote В is used i D d 
Gie., for 87). The R’s thus defined are positive quantities if s; > ©, e 
negative if s; < C. Note that as | т; | increases | Е; | decreases. Then 
ŝu = B, — (B, — CakiR: , 
where a = R;R,/| В.В, | and | R; | > 12: |. S ete 
Equation (24) gives a set of computations expressed entirely 1 AT 
values by means of which an estimate $,; of the scale value of а ene Дд 
may be computed from the scale values of tlie components, assuming 
negative exponential law of value increase. 


(24) 


scale 
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Method of False Position 


i It was also found that the test equations previously developed were not 
sensitive enough to differentiate clearly between the different value laws. 
The method adopted was 


composite value—selecting the 
squares of the differences between t 
for the composite stimuli. For the lin 
forward. For the others no explieit 
made by a successive approximati: 
of False Position (10). A brief acc 


nonlinear equations presented by Karl Pearson. The problem may be stated 
аз follows: 


Given the k-parameter function 

Y; = f(m, +. ES NEL ; т. ух.) 
together with experimental obs 
* , n), to determine the valu 
Only two restrictive conditions 
(1) Given the value of the 
values m,, for each of the par: 
to obtain by mechanical or ot; 

for the dependent variable. 


ervations of the paired values Js 
es of m, so as to minimize У? 
are necessary: 

independent variable z; an 
ameters, it must b 
her means) а corre: 


› Yi G = 1, 
del (y; — Y. 


d a set of arbitrary 
€ possible to compute (or 
sponding set of values Y,, 


For a one-parameter function the Law of Mean Value may be stated as 


dY,, дар Y, = Yo: 
PE 2 EAR Pay 
ать m, — то’ 


or 


=F (m, — m). 


For a k-parameter function the general Law of the М 


1 ean for functio 
Several variables (сї. 9, p. 121) gives ЭНА 
k 
дУҮ,, 
Y, — Ta = > [2 (т,, = m], 
where 


? is an index for the observations of the dependent i 
i and 
variables (i = 1, ... | n), nd independent 


g  isanindex for the 


parameters in the 
0 indicates the initia, 


function (g — ПЕТУ 
l guess for the val 


k) 
ЕД Ц 
ues of the parameters, 


—t 


* 
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h indicates subsequent guesses for the value of the parameters (h= 1, 
e E. 

p indicates that the value of the partial derivative is taken at a point 
p on the curve between mo, and Mas , 

mj, indicates the hth estimate for the parameter m, . 


Pearson's derivation is algebraic and quite voluminous. À much briefer 
statement in terms of matrices has been given by Gale Young in an un- 
published note to the writer. This derivation is presented with acknowledg- 
ment to Dr. Young. We may put the derivation in matrix terminology as 
follows: Let 


Y bea matrix of k rows and n columns with elements Y; — Yo: (ћ = 1, 


eV Ry $m 1, +++ m). 
M be a square matrix with elements m,, — Mos, where (g= 1, =e g fy 
h=1,-+:,k). 


F, be a matrix of k rows and n columns with elements дҮ,:/дт,, . 
Then from the Law of Mean Value 
Y=MF, o M"Y=F,. 

Thus, we have a means of eliminating matrix F, , for which it would be very 
difficult to find reasonable experimental values. In order for M^! to exist, 
M must be a square matrix; hence for k parameters there must be k+1 
guesses for each parameter. Thus, both g and Л must vary from 1 to k. It 
should also be noted that the partial derivatives in P, are taken at some 
suitable point p between то, and m,, , selected so that the Law of Mean 
Value holds. Let 


y; (i= 1, --- ,m) designate the set of observed y values, 
ть, designate the parameter values which give the best fit, 


Y,: designate the corresponding values for the best fitting values of the Y's. 


We may now define the following row vectors: 


m with k elements m,, — mo, designates the correction needed to change 
the first (or zero-th) guess into the best b guess, 

c with n elements У; — Yo; designates the corresponding chani 
calculated Ys, 

d with n elements y; — Yo; designates the difference b 
approximation and the observed values, and 1 

€ with n elements y; — Y,; designates the error of fit for the best уа ee 
The problem may now be stated as follows: solve for the vector m 1n 

terms of Y, M, and d so that ee’ is a minimum. 

From the definition of elements we see that 


ges in the 


etween the zero-th 


NEC E (25) 
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also д 
€ = mF, , (26) 
Where q designates а suitable point on the curve between Mo, and n,, . Thus, 
2 e=d—mF,. 


(27) 


gression problem, which 

g this procedure, 
ee’ = dd’ — dFin' — mF, d! + тР Fim’, 

Differentiating with respect to m and setting 


Selecting m so as to minimize ee' is the multiple те 
is solved as indicated in (16, pp. 173-174). Followin 


2тР,Р; — 24р! = (у. 


If the changes in Parameter values are sli 
equal to F, , then substituting M^'Y for p 


the correction term m expressed in te 


е trial parameters M, Y, and d, the Observed and 
pendent variables, 


rms 
pre- 


f trial param- 
eters Masne, which should шү 


G41): Which is better 
than any of the predictions Y.; to Y,, previously obtained, The new vector 


vector m,, giving the 
eters used to obtain a 


Spondingly, since Y = МЕ, the vari 


сении" 2 
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enough so that a line, plane, or hyperplane is а good fit to the surface in 
the region being dealt with. Under these conditions the equation Y — MF 
will be а good approximation to the surface. Changing only one parameter 
at а time, so that M is a diagonal matrix, would be a simple method of 
satisfying this requirement. 

The Method of False Position with some of its extensions and limitations 
has been discussed by Willers (17). He also presents other methods for solving 
problems of the type considered here. 


Predicting Scale Values of Composite Stimuli 


The four laws (equations 19, 20, 21, and 24) were used to predict the 
scale values of the composite stimuli from the scale values of their component 
stimuli. 

_ For the negative exponential law four sets of values were chosen for 
B , B*, and С. These values were used with (24) to give sets of values for 
Si; . The parameter values were used to construct the matrix M. The values 
of §;; gave matrix Y. Using s,; апа $,, gave the vector d. Equation (30) is 
then used to find a correction which gives а fifth set of values of the parameters, 
which is better than any of the first four. This process is repeated until a 
minimum is found and tested. For a one-parameter system the process is 
similar but much simpler. To give a measure of goodness of fit we have 
presented the sum of the squares of the diserepancies as well as the sum of 
the absolute values of the discrepancies. These values are shown in Table 3. 

It can be seen that the logarithmic and square-root laws in this case 
give the largest discrepancies between the actual values of the composites 
and the values as predicted from the single stimuli. Therefore, we shall not 
consider either the logarithmic or square-root laws in further detail. ‘ 

Both the linear and negative exponential laws placed the zero point 
TABLE 3 


Four Value Laws Compared 


Additive 
Constant 


Negative 


Exponential 

Linear 43h 

Logarithmic .568 
.815 


Square Root 
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between pork and lamb, where previous consideration showed it reasonably 
came. 


Figure 1 shows the test for the linear value increase—the plot of the 


3 


LINEAR VALUE LAW 


[S =(5,+ sp- 9373] 


Sis 


] 
FIGURE 1 


Scale value of the composite against the sum of the scale values of the com- 
ponents. This plot gives a reasonably good fit to the line 


$ = (s; + з) — .9373. 
Thus, the best estimate of the zero point on the assumption of the linear 
law is .9373, or about .94 if we assume that х, = 0. 
A more detailed analysis sho 
is shown in Figure 2. Here we hav 
abscissa, and over it the value 


wing the fit for the negative exponential 
e the value of tongue, .137, plotted on the 


of each of the composites with tongue— 
tongue and pork, tongue and lamb, tongue and beef, and tongue and steak. 


The same has been done for pork, lamb, beef, and steak and their composites 
with each of the other four stimuli. 


The lines show how a negative ex 


ponential rule would fit th 
given that the zero point is at .8, that th 


rge at the two points (4.6, 4.6) and 
(—1.8, —1.8), corresponding to the + 
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NEGATIVE EXPONENTIAL LAW 


FIGURE 2 


are smallest for the negative 


val Я z " 
ue laws even though the discrepancies 
more points in the upper 


ехровенвиА. More points are needed, particularly 
right quadrant, e.g, where both components are positive and hence the 
composite is positive. Similarly, if one has negative values, there should be 
more of them, so that the negative components would combine to form а 
number of different negative composites. It had been expected that all the 
values in this case would be positive and hence that five components might 


have been adequate. 


Summary 

А been developed: à square-root, à log- 
arithmic, a negative exponential, and an additive law. A method has been 
presented for testing each law using only the scale values of components and 
composite stimuli determined by а psychophysical scaling method. In this 
case paired comparisons was used. A tentative extension of these rules has 


Four different value laws have 
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been made to cover the case in Which both negative and positive components 
are combined. 


€ commodity, and in 
which the usual Scaling methods do not give a zero Point. 
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MAXIMIZING TEST BATTERY PREDICTION WHEN THE 
WEIGHTS ARE REQUIRED TO BE NON-NEGATIVE 


em 


ЈоѕЕРН LEV 


NEW YORK STATE DEPARTMENT OF CIVIL SERVICE 

weights 
ximize, 
dictors 


A procedure is developed for computing optimum regression 
under the restriction that they be non-negative. The weights ma 
subject to the restrictions, the multiple correlation between several pre 
and a criterion. A numerical example is provided. 


‚ For some purposes weights of subtests of an examination must be 
Positive. This is true, for example, in civil service examinations when weights 
are announced in advance of administration of the examinations. Positive 
weights are needed in order to motivate candidates to perform well in the 
subtests. The problem considered in this paper is the computation of weights 
of subtests so as to maximize prediction of a criterion, with the restriction 
that the weights be non-negative. 

The procedure to be described is similar to the method of “steepest 
descent" (e.g., 1, p. 47). Actually it is more appropriate to call this а method 
of Steepest ascent, since the weights are determined by successive addition of 
t increments which tend to increase the multiple correlation between the 
predictors and the criterion. ; 

An iterative procedure to accomplish the purpose considered in this 
paper was previously described in (2). Advantages of the present method 
are that the computations at each step in the iteration indicate the procedure 
at the next step, the termination of the procedure is clearly indicated, and the 
Weights obtained are the best possible. 

' In order to write the necessary formulas, 
be adopted: 


the following notation will 


X, — criterion 
X, = predictors (i = 1,2, ++- , 9) 
a; = weights for predictors (¢ = 1 PARU) 7 
T = aX, + „Х« + +: +ыХ„ 2 total weighted predictor score 
C; = >} (X, — X) = sum of squares for X; A 
С. = У(Х, – ¥) (Х, – X,;) = sum of products for X; and X; 
R = correlation between X, and T 


The L,; used by Wherry and Gaylord may be used here in A of 
C;; . The C;,; are smaller numbers than the L,; and are sufficient for 
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accuracy required. In this notation 
by а:С,; 
1 


: (1) 
V Coo Al bs 2;a;C;; 


In (1) R may be viewed as a function of the variables a; ; the С „ате 
constants. Tt is convenient to introduce the total differential 


R= 


aR a ‚у OR 2 
ai= ойи. laH тоа. (2) 


Неге dR is an increment in R, da; are increments in the weights, and 


ðR/ða; are partial derivatives. (2) shows how changes in weights influence 
changes in R. If partial derivatives have b 


the a; , values of the da; can be selected Б 


iterations. 
steepest ascent.” 
atives we note that 

oR ЕТА 

да, 7 Coo D 8,0, 0с, уз а,б], (3) 


ed, only relative values of the partial 
only the quantities within the square 
brackets are computed. 

In essence the computation involves re 
the weights, da; , so that each trial produces 
with the restriction that the 
any trial are determined by thi 
at the previous trial. The tria 


the terminal weights а,; ms of the increments tried 
at the various iterations, F i 


Xe. 
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A justification for the procedure will be discussed more fully after the compu- 
tations have been described. 


Computational Procedures 


The computing procedures will be described first symbolically, and then 
in relation to a numerical application. The symbolie development which 
follows is outlined in Table 1. 

1. Record the sums of squares and cross produets of the predictor and 
criterion variables as a square matrix. 

2. It is helpful, though not essential, to record below this matrix the 
correlations between the predictors and the criterion. 

3. Below the matrix record the increments corresponding to the iter- 
ations in the appropriate columns. АП the increments corresponding to one 
iteration are placed in a single row. The symbol da;; will be used to denote 
the increment for the variable X; in the ith iteration. 

4. After m iterations the weight а; for variable X; is the sum of the 
increments used to this point, 


бш = 25 да;; - 
i-1 

The terminal weight for X; is the a,; after all iterations have been completed. 

5. The first increment is obtained for the variable, X; , which has 
highest correlation with the criterion. It is an approximation of one or two 
digits to the ratio C,;/C;; . Increments for the remaining variables are 
ordinarily, though not necessarily, taken to be zero in the first iteration. 
These increments are also the weights after the first iteration. That is, 
ai; = da; for each predictor X; . 

6. The computations which lead to the next iteration are carried out in 
columns set up at the right of the matrix of sums of squares and cross produet 

7. In the first column of this set-up the following computations are 
made successively: 


5 a;Ci; , 25 aglar a » ai;Csi » 27 ai; Coi › 
х. zs а:а,,С:; = 25 du. аб) (5 = 12, LZ 


i 


ky = (У al E Luat 
Qs = kı Э а: Со: * 


Formula (4) provides a measure of the correlation between the weighted 


sum and the criterion, attained as a result of the first iteration. A similar 


quantity is computed at each iteration. 


(4) 


TABLE 1 


== SYMBOLIC LAY-OUT FOR COMPUTATION ОЕ WEIGHTS, SHOWING TWO ITERATIONS 
ird 3 ES I II 
X *15...0, бт > атуу масс 
x 
e GHI Cn Ф Pap; 2 api 
X с C og Х ар 39 > a2jČj 
го] On ZZ аца; Б вает 
à ^u daln ky = Se 21jC0j ko = ‚= Р 22C0j 
II ад Чад, 22 аууау;С,; > 2 ар;аруС,; 
ky 2 156; k2 2 a9 iC; 
© day datn Cor - 02 99505 ире Еш 
Тоса1 ac] atn 
Regression 
Ка] ка Con = Ку = аууб; Con - k2 Z а, 


weight 


Еа 


YXISRLSKOHOXSd 
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The succeeding computations have as their purpose an evaluation of the 
partial derivatives exhibited in (2). We compute 


Cor = ky Ў а; С 5777 бш — № 35 axis 


For the first column some simplification of the above computations is possible. 
However, the saving is slight and the formulas are stated in full generality 
as they apply to all columns. Note that the partial derivative which corre- 
sponds to the variable having positive weight becomes zero, or nearly so. 

8. The increment for the second iteration is based on the values of the 
partial derivatives obtained from the first iteration. Ordinarily we choose 
the partial derivative which has the highest positive value and select an 
increment for the corresponding variable. If this variable is X, , then the 
increment daz, is chosen so that 


Con E У а,б 2e 


approximately. Here the аг; are the weights after the second iteration. 
Increments for more than one predictor may be chosen at any iteration if this 
seems appropriate. 

9. To perform the computations for the second iteration we take ad- 
vantage of the relationship 


57 asjC 5 = 27 аби + 2 (daz;)Ci; - 


priate for the computation of 


However, this simplification is not appro; 1 
carried out as described 


D У) axa; Ci; . The remaining computations are 
for the first iteration. 

10. The iterations are continued unti 
decidedly negative, or approximately zero, ап 
seems to have attained a maximum. 


1 the partial derivatives are 
d the quantity kp У anjCo; 


= mmu 
Details of a numerical example are shown in Table 2. The first positive 
the quotient 1790/3246. This 


increment, daa = 0.6, is an approximation to i 
increment is also а,» . The computations based on this increment are shown in 
the column headed I. The first seven entries in this column are products of 
the increment .6 and the cross products under Х as .6 (2802) — 1681, =+; 
.6 (1790) = 1074. The eighth entry is .6 (1948) = 1169; k, is 1074/1169 = 
9187; ky У абы = CoR? is given as .9187 (1074) = 986.7. The last six 
entries in this column indicate the relative values of the partial derivatives. 
The greatest of these is the one related to X, and has the value 2520 — 
(.9187) (1654) — 1000. This value of the partial derivative suggests the 
positive increment da з = 0.1 for the second iteration. Computations in the 


c 
z- az D © Юл zy - 905 5060* 0 600" 0 ғау" 0 са 
oez- 1ш- | Lee orz- стт" 0 соо" 0 9 0 [эло], 
I a a zt 0 0 $00* 0 0 0 n 
ozz- līz- | 8т2- 61ё- соо" 0 0 0 0 0 А 
0 0 s- 9t- w 0 0 0 0 0 ш 
sez- oez- | ez- 9sz- flf zy - 105 19 0 0 0 0 0 п 
үшт | "2201 |6'9201 | 0'901 005% хх 0 0 0 0 9 0 ї 
zen" | 0601 | ue: site" 4 
и ил S69T +E9T ШК 961° — 0900'- ФЕР" — coor* 1188 ёг" 1 
69Е1 т | Iset 9zet fü5fe z п 0060 59- 66 ves 06070 нй Ox 
toze £6TE 9276 2662 fogle z ozsz 08 Т Е92Т- 9022 TBST 2822 +621 Sx 
602 161 £02 отг Боз z s9- f97I- 8186 66h ЕШТ 0/8 oze Sy 
ever peat ETZI 1611 90 = 616 9022 66t? 2516 LET тэт £801 ty 
856 1%6 [7 106 89% z 165 Test — шт Hel оос тап от fy 
эш 9900 | zszz vac EE 062 150 05 LI9T тва orze z087 S 
681 вт | 8/81 И Mez D л о ЄТ от z087 scU Ty 
a] AL | ш II 1l eynuroy F Oy 9x Sx fx су [" 
$1H913^ 30 NOIIVIQndHOO 3AI1Y331I 
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column headed II will be given in full. Notice that 


1681 + .1(1794) = 1860 
1948 + .1(2757) = 2224 
769 + .1(1581) = 927 
970 + .1(2206) = 1191 
342 + .1(—1263) = 216 
1654 + .1(13380) = 2992 
1074 + .1(2520) = 1326 


| 
| 


.6(2224) + .1(2992) = 1634 
1326/1634 = .8115 
8115 (1326) = 1076.0 

1254 — (.8115) (1860) = —256 


1790 — (8115) (2224) = —16 
534 — (8115) (927) = —219 
979 — (.8115) (1191) — 12 

—65 — (8115) (216) = —240 


2520 — (.8115) (2992) = 90 


Iterations and computations continue as shown in Table 2. At iteration 
5 the process stops because the partial derivatives for variables X» , Xa, Xe 
are nearly zero and the remaining partial derivatives are negative. Note also 
the stability of CoR? over the last three iterations. The result is a multiple 
R of .2994, with regression weights 0, .4723, 0, .0039, 0, .0905. 

All the increments tried in this numerical example have been positive. 
In some circumstances negative increments may be tried. This is true when à 
positive weight has been too large as shown by а negative partial derivative 
for a variable having positive weight. 

An evaluation of the method is called for at this point. One may ask 
whether the weights obtained by this method actually provide the highest 
possible R under the restriction of non-negative weights. Before dealing 
with this question it is necessary to consider whether there exists а set of 
positive values a; which yield a maximum for R. Р 

One way of demonstrating that а maximum actually exists is to note 
first that under ordinary conditions à set of weights maximizing R can be 
found for any selection of variables out of the n given variables. Some of the 
selections of variables will give à maximum R with non-negative weights. 
There are a finite number of these selections. Consequently, we may choose 
that one set of variables which provides the largest maximum R under the 
required condition. Weights obtained in this way are the desired weights. 
Thus, the existence of a maximum is demonstrated. In exceptional circum- 
stances it may happen that all weights are negative. This will be shown by 
the computation. 


The existence of a maximum R under the condition of non-negative 
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weights having been established, it is necessary to demonstrate that the 


procedure described in this paper actually leads to this maximum. This will 
be demonstrated with the aid of a theorem. 


The necessary and sufficient condition for the set of weights a, , a; RIT 
lo maximize R, with the provision that none of the weights be negative, is that 


the partial derivatives aR/da; be zero for variables X; with positive weights, 
and negative or zero for variables 


al derivatives will decrease 
R. Thus necessity is demonstrated. 


Sufficiency: To demonstrate sufficiency, Suppose that the procedure of 
this paper has provided a Set of a; satisfyin 


ding R. Suppose now that the: 


erivatives. Any change in | 
values of the a; for variables whi i | 


; the procedure provides а unique maximum under the 
restriction to non-negative weights, 
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THE MATCHING PROBLEM* 
EDGAR J. GILBERT 


HARVARD UNIVERSITY} 


Tables of the exact distributions of number of matches are given for 
small decks having the same number of cards in each suit. Several approxi- 
mate distributions are considered for use with larger decks, and some indi- 


cation of the goodness of the approximations is given. 


Many psychological experiments involve testing the ability of a person 
(or а method) to classify certain objects into categories. The matching- 
problem technique to be described in this paper provides a mathematical 


. model which may be used to calculate significance levels for such experiments. 


'The example of the matching problem which is most often cited is the 
work done in certain of the early ESP (extra-sensory perception) experiments. 
A deck of 25 cards containing 5 each of 5 different figures (circle, cross, wave, 
square, and star) was shuffled into a random order by the experimenter. The 
subject was given a second deck of the same composition and asked to arrange ` 
it in the same order as the hidden deck of the experimenter. Then the two 
decks were compared. If the first card of the subject’s deck was of the same 
kind (circle, cross, etc.) as the first card of the experimenter’s deck, the 
subject scored a “match.” Then the second card in each deck was examined, 
and so on through the two decks. The subject’s ability was scored according 
to the total number of correct matches, as compared to the number to be 
expected by chance alone. 

For another example, suppose а handwriting expert claims he can tell a 
person’s profession by examining a sample of his handwriting. To test his 
ability he is given 10 samples, 2 written by doctors, 4 by lawyers, and 4 by 
teachers. He is told which professions are represented but not how many 
samples are from each profession. If we number the samples from 1 to 10, 
placing under each number the true profession and then the expert’s guess, 


we might get the following result: 


Sample Number: 1 2 3 4 
True Profession: L D L T у 
Expert’s Guess: L D T T D T D b D 
*The calculations for thi: hile the writer was working on & project 
Do P funds from the [reg Meere The writer wishes to NE his gratitude 
E this eeu Меен for his constant help and encouragement during Se 
TNow at University of California, Berkeley. 
253 
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In order to evaluate the expert's ability on the basis of this test, we 
might think of the two rows “True Profession" and "Expert's Guess" as 
i all deck, respectively, and 


matching problem similar 
to that of the ESP experiment described above. We notice that he called 


‚ 2, 4„5, апа 6, for a total of 5 correct matches. 
ifferent from the ESP case in that the 


i » Where any given 
‘in the target deck. 
Istory in the mathe- 


—— ee 


Т en л ё 
=н 
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matical literature. For example, Feller (3) gives а formula for calculating 
the probability distribution of the number of matches in this case and gives 
tables of the calculated values for decks of size 3, 4, 5, 6, and 10. He shows 
that for deck sizes larger than 10, the values are, to five decimal places, the 
same as for the Poisson distribution with unit mean. 

A special case of the matching problem arises when there are only two 
suits in each deck and each suit has ¢ cards in it, making a total of 2c cards 
in the deck. There is no possible arrangement of the call deck which will 
give an odd number of matches, while the number of arrangements giving 


h matches, when h is an even integer, is UL. the square of the number of 


combinations of c things taken h/2 at a time. Since this is the square of a 
binomial coefficient, it is a simple matter to look up the binomial coefficients 
corresponding to h = 0, 2, 4, => 2c, square them, and add the results to 
get the total number of ways of arranging the deck. Then divide the number 
of arrangements which give h matches by the total number of arrangements 
to get the probability of exactly h matches. For example, if each suit has 2 


2 2 
cards in it, then there are A = 1 way of getting 0 matches, i = 4 ways 


2 
of getting 2 matches, and H = 1 way of getting 4 matches. So the probability 


of exactly h matches when h = 0, 1, 2, 3, 4 is 1/6, 0, 4/6, 0, 1/6. 

The next easiest case to calculate is that of two identical decks, each 
having s suits of с cards per suit. It is this situation, where the probability 
of exactly h matches is a function of the three numbers: s$, the number of 
suits; c, the number of cards in each suit; and h, the number of matches, 
which will be the primary concern in this paper. Denote this probability by 
m(s, c; h) and write the probability of h or more matches (which is the sum 
of the probabilities for h, h + 1, h + 2, etc.) as M(s, c; h). It is this latter 
probability which is usually wanted in finding the significance of experimental 
results. We usually want to know the probability of a subject’s doing as well 
as or better than his result by chance alone. 

Even in this case, few tables have been published : 
probabilities. Huntington (8) gives tables of m(3, 3; h) and m(4, 4; h). Greville 
(5) gives a table of m(5, 5; h), the probabilities for the ESP experiment 
referred to earlier. Greenwood (4) gives a table of estimated values of m(4, 13; 
h) for values of h from 0 to 7, where both decks are the ordinary 52-card 
bridge decks. There may be other tables available in the literature; but it 
was thought that a small collection of tables in one place might be useful. 

Greville (6) derived a formula for the matching problem distribution 
which seemed to the writer to be more adapted to calculation of exact proba- 
bilities than some others in the literature. His formula was for a more general 
case than that we are now considering and will be used in its general form 1n 


giving the exact 


Exact Distributions. 


TABLE 1 


Probability of h or More Matches 


M(9, 1h) M(10,1;h) ^ M(11, 13h) 


- 04167 


. 00833 


+ 00833 


. 02222 


+ 00139 
‚00139 


. 018. 


+ 00436 
. 00020 
‚00020 


25 


M(2, 1h) MG, TR) —_M(4, lih) M(5, Gh) M6, GA) M(Z,; lih]. — M(8, Ih] 

1. 00000 1. 00000 1. 00000 1. 00000 1. 00000 1. 00000 1. 00000 1, 00000 1. 00000 1. 00000 
50000 - 66667 . 62500 = 63333 263194 163214 163212 63212 63212 ‚63212 
- 50000 - 16667 -29167 -25833 - 26528 126409 126426 26424 26424 ‚26424 

+ 16667 ‚04167 . 09167 .07778 . 08075 . 08023 08031 . 08030 .08030 
-01912 01897 01899 ‚01899 


. 00365 
. 00059 
. 00008 


..00001 


.00001 


200001 


. 00001 


‚00002 


. 00001 


M(3, 3;h) M(4,3;h) ^ M(5,3;h) ^ M(6,3:h) —M(7,3;h) M(8, 3;h) M(2, 4;h) 
1. 00000 1. 00000 1.00000 1. 00000 1.00000 1.00000 1. 00000 1, 00000 
.95000 - 96667 + 96258 -96016 -95852 -95734 95645 ‚98571 
-95000 + 83810 - 82875 - 82266 - 81875 - 81603 ‚81402 ‚98571 
.50000 «61310 . 60010 ‚59456 59113 . 58882 58715 + 75714 
. 50000 «35417 = 35404 + 35335 +35313 -35301 +35295 +15714 
. 05000 + 16131 . 16724 ‚17154 ‚17407 ‚17578 .17702 . 24286 
. 05000 - 04881 . 06322 ‚06808 -07113 ~ 07319 . 07467 . 24286 
-01667 ‚01864 .02215 . 02425 . 02569 . 02674 .01428 
- 00060 - 00461 . 00592 . 00694 . 00766 . 00820 .01428 
. 00060 . 00073 - 00130 - 00167 = 00195 ‚00217 
. 00015 . 00023 . 00034 . 00043 . 00050 
. 00000 . 00004 . 00006 . 00008 . 00010 
. 00000 . 00000 . 00001 . 00002 


М(4,2һ) M(5.2:h] M(6,2:h)  M(7,2:h) — M(B,Z:h] M(9, 23h) — M(ü0,2;h)  Mül, ih) 

22:00000 1, 00000 1. 00000 1.00000 1.00000 1, 00000 1. 00000 1. 00000 1. 00000 1. 00000 
83333 + 88889 -88214 ‚87870 -87632 — ©8746; .87335 .87237 .87158 ‚87094 
83333 . 62222 61548 + 60991 + 60689 . 60483 «60334 . 60220 . 60131 + 60060 
16667 132222 ‚32341 132451 25240608 855585 132369 5560 5521 “Sey 
216667 +14444 + 13294 +13262 -13450 ‚13580 ‚13676 ‚13749 ‚13806 ‚13853 
.01111 103532 - 04179 04396 `. 04538 ‚04640 ‚04716 ‚04776 ‚04824 

“01111 .00992 .01033 .01144 ‚01226 401285 + 01330 . 01365 01394 

‚00040 100177 .00237 .00270 ‚00295 ‚00313 00328 ‚00340 

-.00040 100036 .00041 ^ .00049 - 00056 ‚00062 00067 . 00071 

- 00001 .00005 .00007 - 00009 - 00010 00012 00013 


M(3, 4;h) 
1.00000 
. 99001 
‚93737 
. 80646 
. 60052 


-37117 
- 18831 
.07472 
. 02485 
. 00511 


- 00141 
. 00003 
.00003 


1. 00000 


«98813 
«93005 
«79355 
«58930 


+37156 
+ 19645 
. 08658 
. 03174 
. 00962 


98 


YXIHISNOHOXSd 


4 
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TABLE 1 (Continued) 


h M(5, 4;h) M(2, 5;h) M(3, 5;h) M(4, 5;h) M(5, 5;h) M(2, 6;h) M(3,6;h) M(2, 7:h) M(3, 7;h) 
0 1. 00000 1. 00000 1. 00000 1. 00000 1. 00000 1.00000 1. 00000 1. 00000 1. 00000 1.00000 
H . 98693 + 99603 + 99702 + 99624 ‚99571 .99892 .99911 .99971 .99974 . 99992 
2 . 92558 «99603 . 97700 ‚97286 ‚97026 .99892 . 99186 .99971 .99720 ‚99992 
3 .18646 . 89682 .91278 . 90271 ‚89690 ‚95996 «96331 .98543 .98532 + 99495 
4 .58385 . 89682 .78150 ‚76776 + 76033 + 95996 . 89136 + 98543 + 94969 + 99495 
5 - 37138 - 50000 ‚59102 .58113 ‚51600 . 71645 . 76155 ‚85693 . 87266 ‚93403 
6 .20090 „50000 . 38388 . 38408 . 38406 .71645 .58399 . 85693 . 14532 «93403 
7 .09232 + 10397 .21077 . 21933 .22380 .28355 + 39331 .50000 . 57839 . 69036 
8 . 03609 ‚ 10397 . 09571 . 10764 . 11374 . 28355 . 22923 . 50000 . 40070 ‚69036 
9 .01202 . 00397 .03624 . 04528 . 05043 . 04004 . 11416 . 14306 . 24472 ‚30964 
10 ‚00342 ‚00397 . 01074 . 01630 ‚01953 . 04004 . 04838 .14306 . 13051 . 30964 
11 „00083 . 00281 . 00500 . 00662 . 00108 . 01709 ‚01457 . 06040 . 06597 
12 .00017 . 00043 . 00131 .00196 . 00108 . 00513 . 01457 . 02404 ‚06597 
13 . 00003 . 00010 . 00029 . 00051 ‚00120 . 00029 . 00823 . 00505 
. 00000 . 00005 . 00012 . 00026 . 00029 . 00237 ‚00505 


.00001 .00002 . 00003 . 00059 .00008 
. 00001 . 00011 .00008 
‚00002 


M(2,9;:h) — M(2, 10;h) -M(2,11;h) 


LudsTI9 ‘г HYOGd 


1. 00000 1.00000 1.00000 1. 00000 
‚99998 - 99999 1.00000 1.00000 
.99998 „99999 1.00000 1.00000 
‚99831 .99945 .99983 . 99995 
‚99831 «99945 . 99983 «99995 
+ 97166 + 98849 99554 . 99834 
+ 97166 + 98849 „99554 . 99834 
+ 82653 .91055 .95694 .98044 
‚82653 „91055 ‚95694 ‚98044 
„50000 ‚67186 . 80257 . 88983 
.50000 .67186 .80257 .88983 
«17347 .32814 . 50000 .65186 
. 17347 . 32814 . 50000 . 65786 
. 02834 .08945 . 19743 .34214 

.02834 ‚08945 ‚19743 .34214 
. 00169 . 01151 ‚04305 ‚11017 
. 00169 ‚01151 ‚04305 ‚11017 
. 00002 . 00055 . 00446 ‚01956 
‚00002 -00055 ‚00446 ‚01956 


LSZ 
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the last section of the paper. The following Special case of Greville's formula 
was used to caleulate the values given in Table 1: (n = sc = size of deck) 


m(s,c;h) = I > (= y(n = O10, , (1) 


where Н; is the coefficient of x’ in the expansion of 


= Clcla? * 
(Exc -$6-5 i @) 


Since it is usually the probability of h or more matches which is wanted 
in checking experimenta] data, Table 1 lists M(s, c; № = Sor, m(s, c; i). 


The calculations were carried out to exact integers before dividing by nl, 


of M(s, c; h) rounded to 
st by groups in order of 


for M(2, c; h)] has the 
even h. This is due to + 
case of exactly h matches, 

The values M (5, 5; 


Approximate Distributions 


Since the calculation of exact probabilities Sets very tedio; 
decks, it would be desirable to b istri 


already available 


mation of m(s, 1 ; h) 
tributions within each gr 


2 —À ———PMÓ — 
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then in terms of c, the number of cards per suit, and n = sc, the size of the 
deck, we have 


и = с 
pe = e(n — ©)/(п — 1) 
_ (3) 
ш = сп — O(n — 2c)/(n — 0 — 2) 
m c(n — с) 
H= (п = 0 = 2) – 3) he 90 SO 


+ (c — 1)(12ne — n — 18c — бо)]. 


Anderson (1) shows that as the deck size is increased while keeping the 
proportion of cards in each suit fixed, the number of matches is asymptotically 
normally distributed. So if n is large enough, the mean and variance of the 
matching problem distribution can be calculated from formulas (3), and the 
desired values found in a table of the normal distribution. Unfortunately, it 
18 difficult to say precisely when n js large enough. In many cases, the match- 
ing problem distribution is asymmetrical; to the extent that this is s0, the 
normal will give a bad fit. Although the normalis not as close an approximation 
as some of the distributions considered later in this section, it can be seen 
from the examples in Table 2 that it is sufficiently close for some purposes; 
it has the advantage of being available without too much calculation. An 
example of the calculations involved in fitting a normal to a matching problem 
is worked out in the last section of this paper. 

Hamilton (7) noted that the matching problem distribution was some- 
what like a binomial distribution with p = 1/s and n = sc, and suggested 
that writing the mean of the matching problem as пр and variance as 
прд{п/ (n — 1)] made the similarity apparent. Tt can be seen that the binomial 
with p — 1/s has the same mean as the matching problem, and that the 
ratio of the two variances approaches 1 as т gets large. This is also the case 
with the third moment about the mean. It is npq@q — 1) for the binomial, 
and we can rearrange the formula for из in (8) above—substituting np for 
c where it appears and setting т = 1 — p—to make the third moment of the 
matching problem look like npq(2q — 1) /(% — 1) (n — 2). { 

The binomial is suggested as a good approximating distribution for 
another reason—extensive tables of the binomial are available (11). Since 
the binomial is а two-parameter distribution, we сап in many cases find 2 
tabled distribution which has both mean and variance equal to that of the 
matching problem distribution. In this case, We shall not look up the same 
values of n and p used in the last paragraph; we shall treat n' and p’ simply 
as two parameters, using the prime to show that these are different numbers. 


If we replace n in (3) by sc and equate the means and variances of the match- 
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TABLE 2 


Approximate Distributions 


-C is Gram-Charlier approximation; 
Ham of Gram-Charlier, Values of M(4,13;h) and its 2d order 
G-Care taken from Greenwood (4), summed to four decimal Places. 
The binomial approximation to M(4, 13;h) was Chosen by method of 
text to have same mean and variance as exact distribution, 


G-CB is the binomial modi- 


a 


Poisson Binomial 4th Order 


0 1.00000 1. 00000 1. 00000 1. 00000 
1 + 87335 - 86466 + 87564 + 87330 
2 + 60334 + 59399 + 60528 - 60340 
3 + 32369 + 32332 + 32317 + 32361 
4 - 13676 - 14288 - 13509 + 13671 
5 + 04640 - 05265 + 04514 + 04643 
6 + 01285 + 01656 +01229 ‚01287 
7 + 00295 ‚00435 +00277 ‚00295 
8 + 00056 + 00110 + 00052 . 00055 
9 + 00009 + 00024 + 00008 . 00009 
0 . 00001 . 00005 б 


Poisson 


Binomial 
M(8, 3;h) У а 3 


n= 


0 1. 00000 1.00000 1.00000 1.00000 
1 -95645 «95021 +95761 ‚95646 
2 + 81402 + 80085 + 81630 81400 
3 + 58715 + 57681 + 58865 58712 
| + 35295 + 35277 + 35256 35295 
5 +17702 ‚18474 ‚ 17549 ‚17704 
6 + 07467 + 08392 + 07319 07468 
1j + 02674 + 03351 + 02583 + 02674 
8 + 00820 +01191 + 00778 + 00820 
9 + 00217 - 00380 + 00202 00217 
10 + 00050 - 00110 ‚00045 00050 
1 00010 ‚00029 - 00009 00010 
l2 . 00002 . 00007 . 00002 


2nd Order 


1.00000 1.00000 


9838 -9839 +9839 964 
+9149 +9149 +9149 902 
+7707 +7705 +7708 780 
+5725 +5723 +5726 + 602 
+3712 +3712 +3713 +398 
+2099 +2103 +2102 

‚1047 . 1045 


ing problem distribution and the binomial, 
У = с = np! 


ца = oc (s — l/(c—1) = x 


a pair of simultaneous equations which we ca 


— 
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s and c. For solutions 


a! = c(sc — D/(c — 1) (4) 
р = (c — 0/6 — 1. 


Note that n' is not the size of the deck but is just a parameter for which 
we have solved. It may or may not be an integer. If it is, and p' is a value 
which can be found in а table, then the binomial distribution with parameters 
n’ and p’ will have the same mean and same variance as the matching problem 
distribution with s suits of c cards each. If n’ is not an integer, the integer 
nearest to n’ can be used, call it n”, and using p" = с/т"! gives à binomial 
with the correct mean and very nearly the correct variance. The writer 
would guess, based on trying а few cases, that for decks larger than 25, one 
should be able to fit а binomial by this method to about three decimal places. 
Table 2 shows some examples of the use of the binomial as an approximation. 

This approach suggests looking for known, or easily calculated, distri- 
butions which have the first two, three, or more moments in common with 
the matching problem distribution. Greenwood (4), following a suggestion 
by Mantel, used the Gram-Charlier series type B as an approximation. The 
Gram-Charlier uses the Poisson as a first approximation to the desired 
distribution; suitable multiples of the first, second, and successive differences 
of the Poisson are added to correct for the mean, second moment, etc. If 
p(h) is the Poisson probability of h, then the first difference, Ap(h), is defined 
to be p(h) — p(h — 1). The second difference, A°p(h), is the first difference of 
the first difference, i.e., Ар(ћ) — Ap(h — 1), and so on for higher differences. 
The reader will note that Ap(0) is not well defined yet, since we must know 
what value to use for p(—1)—this is not ordinarily found in a table of the 
Poisson distribution. For purposes of the Gram-Charlier series, p(/) is defined 
and equal to zero for all negative values of h. We are now ready to define 
the Gram-Charlier series. Where m(h) is the distribution being approximated, 

m(h) = p(t) + aAp(h) + агд°р0) + asd") + `7. (8) 
e, only the first few terms 


| 


In principle, this is an infinite series; in practic 
are used. The constants a; appearing in (5) are determined by the moments 
of the matching problem and of the particular Poisson distribution used. 
The Poisson is usually chosen so that it has the same mean as the distribution 
being approximated; this makes the value of а, 2610. If и, is the ith moment 
about the mean of the matching problem distribution and m; the correspond- 
ing moment for the Poisson, the next three constants for the Gram-Charlier 
type B are given by 

2а» = u; — m, 
(6) 


баз = 6a; — (us — ma) 
24a, = 36a, — 2а„(бт» + 7) + (us — ma). 
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As an example, Suppose we wish to find an approximate value for 
m(7, 2; 3), the probability of exactly three matches when both decks have 
Seven suits of two cards per suit. (From Table 1 we see that the correct value 
is .32383 — .13580 — 0.18803.) As a first approximation, we look up the 
Poisson with mean 2, and find P(3) = 0.18045. This value is 0.00758 too low. 


For the next approximation, we need to know the variances and the value 
of A*p(3): 


ро = chn — ¢)/(n — 1) = 2014 — 2)/(14 — 1) = 24/13 
M: = 2 
в = 1(24/13 — 2) = —1/13 
Ap(3) = p(3) — p(2) = 0.18045 — 0.27067 — — 0.09022 
Ap(2) = p(2) — Р(1) = 0.27067 — 0.27067 
A'p(3) = Ap(3) — Ap(2) — —0.09022 


0.00000 


econd approximation p3) + ад°р(3) = 0.18739, 
ished a still closer approximation, 
аз and a, . An example of a fourth- 


mations may be obtained by a modi 
which uses the binomial instead of th 


been the writer’s 
istribution when c is 


— Г ТН 


4 


= 
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subject or trial are simpler for identical decks. Also, if each subject selects 
his own categories, there may be a different set of caleulations necessary for 
each subject; whereas if all subjects use the same deck, only one distribution 
need be calculated. Finally, while the writer knows of no investigations of 
the power of this type of test, it seems clear that the subject has a better 
chance of showing his ability, if he has any, when both decks are the same. 
Since, in using the matching problem technique, the caleulations are based 
on the decks actually used, there is no way to give credit for choosing the 
proper categories. 

The calculations are also somewhat more complicated if different cate- 
gories contain different numbers of objects. However, in many cases this 
is not a part of the experimental design which can be changed. For small 
decks, the exact distribution may have to be calculated. If so, the following 
procedure, taken from Greville (6), may be used. (This also works for caleulat- 
we non-identical decks.) Suppose the two decks have the following compo- 
sition: 


Suit 1 2 3 ame $ Total 
Cards Call Deck m D ms т. n (0) 
Per Suit Target Deck m Ta na ns n 
Let М; be the smaller of т; and n; for each ? = 1, 2, ..-. ‚в. Then the prob- 
ability of exactly A matches, m(A), is given by 
m(h) = L » э "(Дв — )!Н,, (8) 
ml 1 h 


where Н; is the coefficient of x’ in the expansion of 


А a т\пк“ | 
П [= kim; — k) (n: — E)! à 


А Аз an example, suppose we wish to calculate the distributi 
in the case of the handwriting expert given at the beginning of the paper. 
The two decks have the composition: 


on of matches 


Suit 1 2 E: Total 
Cards Call Deck 5 2 3 m 
Per Suit Target Deck 2 4 4 10 


is point. Although 
ble: 2 in the first 


We notice one peculiar feature of non-identical decks at th 
both decks have 10 cards each, only 7 matches are possi 


Bursu: Ednl. 


Dated 


E 
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suit, 2 in the second, and 3 in the third. Thus, we will expect to get non-zero 
e 
values of m(h) only for h = 0, 1, --- ] 7. 

First expand the generating function (9) to get H; : 


(1 + 10x + 20:51 + 8z + 122”)\(1 + 12x + 362° + 242°) 
7 
= 1 + 302 + 3642” + 229623 + 8064z* + 156482° + 15360x° + 57602’. 
Then calculate the factor 0; = (n — J)!H; for each j = 0, 1, «59 ,7 
7 . 
Next calculate n!m(h) = >> сэ; Q; , for each h = 0, 1, ·.., 7, 
inh 


where (0) їз а binomial coefficient, which can be obtained from tables. 


With a table of binomial coefficients and the values of Q 
caleulated, this sum сап 


Finally, divide each value by n! to get m(h), the probabili 
matches. In order to get the probability of h or more 


TABLE 3 


Handwriting Expert Example 


h Exact Normal 


Binomial 

0 1.00000 0.9922 1.00000 
1 «96984 +9582 - 9718 

2 +84762 -8508 ‚8507 

3 ‚62064 +6368 +6172 

4 +35556 +3632 +3504 

5 „15238 .1492 +1503 

6 «04444 .0418 ‚0414 

7 «00952 .0078 .0106 


M eor. D LBS ANB 


To approximate this distribution, using either the normal or the binomial, 
first caleulate the mean and variance. For this 


extends the matching problem to 
more than two decks, with the Possibiliti 
matches, ete. Here, only his results for t; 


` — t 
OE —! 
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If the two decks have the composition specified in (7) above, then the 
mean number of matches, v, and the variance, c^, are given by: 


„йы i p» min; 
(10) 


2 iH Р 
= Ha = 10 [С> mn) —n DE maim; + nj) + n? D man). 
slightly from the form in which Battin 
The products mini and mn; 
ble (7) giving the composition 
ightforward. We shall outline 


These formulas have been rearranged 
listed them, for convenience in calculating. 
(т; + т) ean be obtained quickly from the ta 
of the deck, and the rest of the calculation is stra 
the calculations for the handwriting example. 


i 1 о 18 | Total 
^ ms 5 2 3 10 
ы ni 2 4 4 10 
mins 10 8 12 30 
m; + ni г? 6 A 
mini(ms + ni) 70 48 84 202 
v = 30/10 = 3 


w 


о? = [1/100)(9)](30* — 10(202) + 100(30)] 
c? = 1880/900 = 2.0889 


c = V 2.0889 = 1.45. 


Since the normal is a continuous distribution, while the matching 
problem distribution is discrete with probability “concentrated” on the 
integers, we must make a correction when using the normal as an approxi- 
mation. That is, instead of the probability that X = h, we want the prob- 
ability that X is between h — 1/2 and h + 1/2; if we want the probability 
of h or more matches, we must find Prob(X > h — 1/2), where X is normally 
distributed with mean 3 and standard deviation 1.45. (In the special case 
where each deck has only two suits, the probability is concentrated on the 
even integers, and we must subtract 1 instead of 1/2 in order to get the 
proper correction.) Since we have available & table of the distribution of 
Y, a normal variate with zero mean and unit standard deviation, we look up 

Sa 2) 
Prob( Y БУЛЕГЕ 


The values found for h = 0, --- , 7 are listed in Table 3. 

The last entry in Table 5 gives the binomial approximation to our 
matching problem. The binomial which came closest to fitting was that for 
which n = 10 and p = .3, which has mean 3 and variance 2.10. 
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It is shown that Estes’ formula for the asymptotic behavior of a subject 
under conditions of partial reinforcement can be derived from the assumption 
that the subject is behaving rationally in a certain game-theoretic sense and 
attempting to minimax his regret. This result illustrates the need for specifying 
the frame of reference or set of the subject when using the assumption of 
rationality to predict his behavior. 


Learning theory and game theory (together with the closely related 
statistical decision theory) purport to provide theories of rational behavior. 
Implicit in any theory of learning is a motivational assumption that learning 
consists in the acquisition of a pattern of behavior appropriate to goal 
achievement, need reduction, or the like. In parallel fashion game theory and 
statistical decision theory are concerned with discovering the course of action 
in a particular situation that will optimize the attainment of some objective 
pay-off. 

In order to gain a better understanding of the concepts of rationality 
underlying these two bodies of theory, it would be interesting to construct а 
situation in which predictions made from these theories could be compared 
and then checked against experimental data on actual behavior. One situation 
of this kind received considerable attention at the Santa Monica Conference 
on Decision Processes (2, 3, 4). The experiment is one involving partial ге- 
inforcement. At each trial the subject chooses between two alternatives. 
Each alternative is rewarded on a certain per cent of the trials in which it is 
chosen (the trials rewarded being randomly determined) and not rewarded 
on the remaining trials in which it is chosen; the per cent of rewarded trials 
is in general different for the two alternatives. The learning theory advanced 
by Estes provides a prediction as to the frequency (in the limit as the number 
of trials increases) with which the first alternative will be chosen in preference 
to the second (2). The same frequency is predicted by the Bush-Mosteller 
theory when certain assumptions of symmetry are made with respect to tho 
parameters that appear in their model (1, ch. 8). Estes reports several 
experiments that confirm predictions from his theory. 

When this experimental situation was described to a number of game 
theorists at the Santa Monica conference, they pointed out that à rational 
individual would first estimate, by experimenting, which of the two alterna- 
tives had the greatest probability of reward, and would subsequently always 
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select that alternative which would not be predicted by the Estes theory. 
Flood has defended the choices predicted by the Estes theory against the 
charge of irrationality, basing his defense on two points (4, p. 288): 


(a) The proper definition of payoff utilities would be uncle: 


mixed strategy rather than a pure one, for the latter would give it no way to discover any 
pattern effect. 


In the next section, by combining in an appropriate fashion the two 
considerations advanced by Flood—that is, by assuming (a) the subject is 
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value of p, as the 


learning model, Estes (2) predicts that the a: 


Symptotic 
number of tria]s increases will be DAS 
uem l-r 3 
аа гара, o, 
This value for pt may be obtained as the Steady state of the stochastic 
- process 
P1 = 00), where p. ( ADEL 3 Q) 
(1 = т) 72 
for 
РАС + 1 E т\р\(#) 3E (1 Er. т›)[1 к 100], (8) 


HERBERT А. SIMON 269 


so that, if p,(£ + 1) = pil) = P7, 
Q—-a+ti- т)рї = 1— 73); (4) 


from which (1) follows immediately. 

We see that in Estes’ theory ту is the probability that А, will be rewarded; 
but it is also the asymptotic probability that, having chosen А, on а given 
trial, the subject will choose it again on the next succeeding trial. A similar 
interpretation can be given to 7» - Hence, we may interpret mı and 7; as 
the conditional probabilities of persistent behavior when the subject has just 
chosen A, or Az , respectively; while (1 — т) and (1 — тз) are the corre- 
sponding conditional probabilities of a shift in behavior. 

For specificity, let us consider the case where ту > 72 - 'Then the game- 
theoretical objection to regarding as rational the asymptotic behavior 
predicted by the Estes model is that the subject could increase his expected 
reward by always choosing A, . For then the expected reward would be 


m>pim+(- рт (5) 


where the terms on the right-hand side of the inequality are easily seen to 
be the expected reward for the Estes model. 

But the rationality of this game-theoretical solution is compelling only 
under the assumption that the reward probabilities are known to the subject, 
and known to be constant. These are the assumptions that Flood challenges. 
Let us consider an alternative set of assumptions which, while not the only 
possible such set, has some plausibility. 

(i) The subject takes as given and fixed Шет corresponding to the 
alternative he has chosen on the last trial. That is, he assumes the probability 
of reward to be m, or ть , if he persists in choosing again А, or A, , as the 
case may be. 

(i) The subject expects that if he shifts from the alternative just 
chosen to the other one, the probability of reward is unknown and dependent 
on a strategy of nature. $ А ue 
. . Qi) The subject does not wish to persist in his present behavior if there 
is à good chance of increased reward from shifting. He measures his success 
on each trial not from the reward received, but from the difference between 
the reward actually received and the reward that could have been attained 
» he had outguessed nature. In the terminology of L. J. Savage, he wishes 
to ogee his regret. 

_We may formalize : tions 
subject chooses аш ун (ii) shifting his choice. If he persists, 
he is rewarded with probability т (where т = 71» or т = T2 depending on 
whether the previous choice was Аз or 42 › respectively); irrespective of the 
strategy adopted by nature. Erde shifts, be wil receive a reward of 0 if 


nature adopts her strategy. (a), and а reward.of Lif nature adopts her strategy. 


as follows: On each trial, the 
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select that alternative which would not be predicted by the Estes theory. 
Flood has defended the choices predicted by the Estes theory against the 
charge of irrationality, basing his defense on two points (4, p. 288): 


(а) The proper definition of payoff utilities would be unclear 
game-theoretic arguments to a real case, and there is a 
rationalize the reported behavior. Thi 


in attempts to apply 
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so that, if pt + 1) = ma = pi, 
1 = т +1 — тг)рї = (1 — т), (4) 


from which (1) follows immediately. 

We see that in Estes’ theory т is the probability that A, will be rewarded; 
but it is also the asymptotic probability that, having chosen A; on а even 
trial, the subject will choose it again on the next succeeding trial. A similar 
interpretation can be given to Tə - Hence, we may interpret т; and 72 as 
the conditional probabilities of persistent behavior when the subject has just 
chosen A, or A, , respectively; while (1 — ту) and (1 — т») are the corre- 
sponding conditional probabilities of a shift in behavior. 
ni For specificity, let us consider the case where ™ > m: . Then the game- 

eoretical objection to regarding as rational the asymptotic behavior 
predicted by the Estes model is that the subject could increase his expected 
reward by always choosing А, . For then the expected reward would be 


m>pint(l— pdms (5) 


where the terms on the right-hand side of the inequality are easily seen to 
be the expected reward for the Estes model. 
heer rationality of this game-theoretical solution is compelling only 
ok ae ahem that the reward probabilities are known to the subject, 
ae to be constant. These are the assumptions that Flood challenges. 
us consider an alternative set of assumptions which, while not the only 
possible such set, has some plausibility. 
(2) The subject takes as given and 
alternative he has chosen on the last trial. T 
of reward to be т, or т, if he persists in choosing again А: 
case may be. 
"t (її) The subject expects that i 
sen to the other one, the probability of reward is un 
оп a strategy of nature. А 
Y dert Ше subject does not wish to persist in his present behavior if there 
B ROM 15 pie of increased reward from shifting. He measures his success 
d MS 3 not from the reward received, but from the difference between 
c Mn actually received and the reward that could have been attained 
to res DOES nature. In the terminology of L. J. Savage, he wishes 
egret. 
beris ed formalize ‘these assumptions as foll 
es between (7) persisting or (ti) shifting 


he is re : 

her. ae a with probability 7 (where т = m» or т е 

strategy ad previous choice was Аз ог A2 ; respectively), irrespective of the 
adopted by nature. If, he shifts, he will receive & reward of 0 if 


nati 
ure adopts her strategy. (а), and a reward of 1 if nature adopts her strategy, 


fixed Шет corresponding to the 
hat is, he assumes the probability 
or A, , as the 


f he shifts from the alternative just 
known and dependent 


óws: On each trial, the 
his choice. If he persists, 
= ma depending on 


270 PSYCHOMETRIKA 


(8). The payoff matrix corresponding to these assumptions is: 


(œ) (8) 


G) 
(ii) 


where rows correspond to the subject’s strategies and columns to nature’s 
strategies. Regret is defined as the diff 


erence between the actual payoff for 
a given pair of strategies [e.g. (i), (8)], and the payoff that could have been 
realized, if the strategy actually employed by nature had been anticipated 


[e.g., (2), (8)]. Performing the indicated subtractions, the regret matrix is: 


(i) 


(This was obtained from t 
the largest element in the 

Now let p be the 
persists, и be the probab 
regret will be 


he first matrix by subtractin 
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The conditions that the regret be minimum (strictly, minimax) are 
given by 
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Using the second of these equations, we obtain from (6) 
0= е – 1) - (1 — gy, 8) 
whence 
б=т. (9) 
Hence the subject would 


n persist with probability т and shift with 
probability (1 — т). But this is precisely the postulate contained in (2). 
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Hence, we have shown that the behavior predicted by Estes’ theory is identical 
with that which would be exhibited by а rational subject intent on minimaxing 
his regret. 


Comments on the Derivation 


We need not try to decide whether the subjects who behave in conformity 
with the predictions of Estes’ theory are minimaxing regret, or whether they 
are simply behaving in the adaptive fashion implied by the usual learning 
mechanisms. Most economists and statisticians would be tempted to accept 


the former interpretation, most psychologists the latter. It is not immediately 
would provide evidence for 


obvious what source, other than introspection, 
deciding the issue. 

Perhaps the most useful lesson to be learned from the derivation is the 
necessity for careful distinctions between subjective rationality (i.e. behavior 
that is rational, given the perceptual and evaluational premises of the subject), 
and objective rationality (behavior that is rational as viewed by the experi- 
menter). Because this distinction has seldom been made explicitly by 
economists and statisticians in their formulations of the problem of rational 
choice, considerable caution must be exercised in employing those formu- 
lations in the explanation of observed behavior. 

To the experimenter who knows that the rewards attached to the two 
behaviors A, and A, are random, with constant probabilities, it appears 
unreasonable that the subject should not learn to behave in such a way аз 
to maximize this expected gain—always to choose A; . To the subject, who 
perceives the situation as one in which the probabilities may change, and who 
is more intent on outwitting the experimenter (or nature) than on maximizing 
expected gain, rationality is something quite different. If rationality is to 
have any meaning independent of the perceptions of the subject we must 
distinguish between the rationality of the perceptions themselves (i65 
whether or not the situation as perceived is the real situation) and the 
rationality of the choice, given the perceptions. 
_ M we accept the proposition that organismic behavior may be sub- 
jectively rational but is unlikely to be objectively rational in а complex 
world then the postulate of rationality loses much of its power for predicting 
behavior. To predict how economie man will behave we need to know not 
ү that he is rational, but also how he perceives the world—what alternatives 
he sees, and what consequences he attaches to them (5). We should not 
jump to the conclusion, however, that we can therefore get along without 
the concept of rationality. While the Estes model predicts the behavior of 
naive subjects under partial reinforcement, We observe (3) that persos 
trained in game theory and placed in t ally learn to 
choose A, consistently. It appears simpler to postu 


a change in the perceptual model—rather than to attempt to exp 
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behavior in terms of simpler learning-theoretic models..If anything was 


learned during the series of trials b 
it was the appropriate perce 
once that model is assumed. 


5. 


y the subjects who were game theorists, 
ptual model and not the appropriate behavior 
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ches have been used to estimate communalities 


At least four approa 
amian and with minimum 


that will leave an observed correlation matrix R Gr 
rank. It has long been known that the square of the observed multiple- 
correlation coefficient is a lower bound to any communality of a variable of 
R. This lower bound actually provides a “best possible" estimate in several 
senses, Furthermore, under certain conditions basic to the Spearman- 
Thurstone common-factor theory, the bound must equal the communality 
in the limit as the number of observed variables increases. Otherwise, this 


type of theory cannot hold for R. 


I. Introduction 


One of the intriguing problems of factor 
formula for communalities that will minimize 
correlation matrix R. More explicitly, the pro 
matrix U such that R — U^ is Gramian and of minimum rank. 

Let n denote the order of R (and of U), and m the minimum rank for 
Gramian R — 0°. At least four approaches have been used to estimate 


communalities that will yield m: 


analysis has been to find a 
the rank of an arbitrary 
blem is to find a diagonal 


(a) trial-and-error exact formulas 

(b) exact formulas for special cases of R 
(c) successive approximations 

(d) lower bounds. 


The main thesis of this paper is that, in certain senses, the last-mentioned 
of these four approaches provides ''best possible" estimates of communalities 
for an arbitrary R, even though biased in general by being underestimates. 

leaves R — U^ Gramian 


Let u; be the jth diagonal element of any U that ; 
(whether or not with minimum rank), and let h; be the corresponding com- 
munality: 


Milos (oir () 


Let p; denote the multiple correlation coefficient of the jth variable. in R 


on the remaining n — 1 variables, and c; the ‹ corresponding sta! 
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of estimate (assuming all observed variables to have unit variances): 


A= wf Q(Qe1 34. (2) 
'Then it has been shown (2, 927; 3, 293) that always 
р: ж (j-1,2,-..,n. (3) 
No better general lower bound to hi has yet been established than pj. 

We shall prove here that there 
which the equality in (3) holds for n 
all but m of the p; are actual communalities (the remaining communalities 
equal unity). Such matrices R, however, are of a very restricted type. 
at we shall establish applies to the 


as n — œ, then all except possibly 


a given R, then the р? must almost always 
h; when n is large. (Conversely, if the appr 
the Spearman-Thurstone hypothesis of a limited number of common factors 
must be false.) 

Ап even more general 
т to n. If there is to be 
yield the uniqueness м? , th 


2 
i i (or between i 
and hj), the larger the possible differe s, 


nce between alternative “unique” 
h observed variable of R. 


ower bounds p? 


Before going on to our new results, it may be helpful t 
four approaches listed above. 


(a) Trial and Error 


will be established. 
o review briefly the 


t, of course, that U? 
for the same R may 
can lead again to an 


ды: ional form in general. 
mizing U? may occur. 


— 
Uw—— —. 
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(b) Special Exact Formulas 
Some special cases of R make possible exact and rational formulas that 


need no apparent resort to trial and error. The known cases are for m < n/2, 
the most celebrated being Spearman's where m — 1. Thurstone has summa- 
rized a number of such formulas (8, ch. XIII). A caution should be added to 
Thurstone's discussion to the effect that not all the apparent solutions may 
yield Gramian U* nor leave R — 0° Gramian. Actually these formulas beg 


the question, for it is generally not known in advance whether or not m < n/2. 


A specialized formula in effect must be tried on the given R to see if it works. 


Use of specialized formulas thus seems to be but a modified type of trial and 


error. 


(c) Successive Approximations 


Attempts have been made to avoid a direct exact solution for U* by 
tions. An approximation 


taking recourse instead to successive approxima’ 
Ui is guessed, and R — U? is “factored” until residuals are considered small 
enough, leading to a second approximation U? , etc. It has been claimed that 
ius а procedure generally converges to а satisfactory 0° (cf. 8, р. 295). 

lgebraie proof of such convergence has never been published to our knowl- 
edge. For many iterative processes, the value to which convergence takes 
place depends on the initial trial value. That this may be the case for the 
оз procedure seems evident when one recalls that there are many corre- 
ation matrices which do not have а unique set of communalities. Also, 
unless proof is given to the contrary, there is no reason to believe that suc- 
ү егде to some U where R — U? is not 
of minimum rank, if convergence takes place at all. 


The issue of successive approximations is furth 
elihood solutio 


er beclouded by sampling 
п seems the most 


as Rao points out (7). To attain precision 


in the sampling theory, apparently some restrictions have been introduced 


to the nature of the population R, else the possibility of equally minimizing 
alternative solutions would remain. Again, it is not clear when a given Ё 
is valid in practice. 


Qm these restrictions or when the sampling theory 
RR the above was written, the writer received а сору of reference (1) in 
ch a numerical example is given of the failure of Lawley’s iterative рго- 


cedure to converge properly.] 


(d) Lower Bounds 


If we again ignore р А 
F sampling and rounding-off errors, 
to eamin useful lower bounds to communalities * i 
ue and without any hypothesis about or restrictions on R. 1 е 
— pounds. thus far established are the pi» according tO inequality (3) 
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above. It is often more convenient to diseuss uniqueness rather than com- 
munalities, or to use inequality (4) rather than (3): 

cz c (i 2,4). (4) 

Ап important feature of the bounds in (3) and (4) is that they hold 

whether or not there is a multiple solution for U^; they ‘hold for all possible 

solutions simultaneously. Indeed, they lead to a criterion for choosing among 
multiple solutions, as indicated in the next section. 


II. Relationship to the Determinacy of Unique-Factor Scores 

Let т; denote the multiple- 
variables of а unique-factor var 
u; . It has been shown in (6) that 


correlation coefficient on the n observed 
iable hypothesized to yield the uniqueness 


A M еа, с, (5) 


) с 5 7, it suggests that when а choice is necessary 
that which makes the Inequalities (4) as small as Possible is most desirable; 


is fixed for j, so that such a choice 


(C= aL) Gee 0), 


r= 27 1 
. H (6) 
is the minimal cor 


where r; is given by (5) and r* relati 

i é i relation always i 
between two alternative Sets of scores for the Same unique fis Iv 
thesized to underlie uj. [According to (6), ifr? = 5, then r* = Q or alter va 
Score solutions for the same jth unique factor always exist ‘that SAU. 


zero with each other. Even if 7? is as lar, is rai 
у i Be as .9, this r * 
equation parallel to (6) holds for common factors.] PLUS io NE 


III. The “Best Possible 


possible, According to (5) this would impl 


the rj ; there is no apparent way of getting s 


LOUIS GUTTMAN 211 


universally systematic manner. The situation seems to be the reverse: 7; is 
determined by иў rather than conversely. 

The rest of this paper will be devoted largely to showing that (4) is 
actually a “best possible” inequality in the sense that the phrase “ез 
possible” is usually used mathematically for inequalities. The essential 


characteristics are that (a) many correlation matrices R exist for which the 


equality in (4) is actually attained at the same time that minimum rank m 


is attained, and (b) the inequalities in (4) must tend to equalities as n increases, 
under certain general conditions important to the theory of common-factor 
analysis. The bounds improve systematically in general as n increases, ог as 
there is more information available from more observed variables. Further- 
more, inequality (4) leads to inequalities for m that are also “best possible," 
and is closely related to the problem of estimating individual scores on the 
unique factors without any rank assumptions, via image analysis. 

In virtually all attempts to solve the communality problem—whether 


exactly or by successive approximations—the problem is stated as for a 
umber of n observed variables. 


fixed and finite л, or where Ё is from a finite n 
It seems appropriate to ask also what happens to communalities as 7t in- 
creases or decreases. 

While this issue is not discussed very explicitly by most writers, it 
usually seems implied that if the additional variables retain the same general 
kind of content as the initial ones, communalities of the initial ones should 
remain constant for all n sufficiently large. This would imply that for n 
small enough we should generally have m > n/2, or easy exact computations 
for U* (even ignoring sampling error) should be the exception rather than the 
rule. Having m > n/2 for relatively small n does not preclude m from re- 
maining constant—and hence becoming relatively small—as 7" increases. 
It does imply that multiple solutions should be quite prevalent for finite n 
2 practice. Furthermore, it cautions that an apparently exact solution for 
finite n may be but an artifact due to the finiteness 0 
observed. 

ы It would be desirable, in view of all the pre 
ave a systematic way of getting information abou 

UM whatsoever about R, yet without resorting to trial and error. 
urthermore, this information should remain valid as n increases. 

One of the virtues of the bounds (3) and (4) is that they possess these 
а in а simple and direct manner. This seems to be another type of 
best possible” property from that usually considered, and one which appears 


peculiarly relevant to the problem of factor analysis. 


{ the number of variables 


ceding considerations, to 
t communalities with no 


Гу; Attaining Equality When n Is Finite 


2 
Tf o? = 0 for some j (so that p? = 1), then it must be tha 
(4) and the fact that a uniqueness cannot be negative. Here 18 one 


at u? = 0 from 
kind of 
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special cireumstance wherein our bound becomes an exact estimate even 
when m is finite. In practice, this is not to be expected, since having one 
observed variable perfectly predictable from all the rest makes R singular. ) 

Many cases of nonsingular R also exist for which the equality in (4) 
holds and т is finite. We shall exhibit some now. To this end, let us first 
recall that the oj are the reciprocals of the corre: 
of R^. The following notation will be useful her 
(the inverse of S^) be the diagonal matrix with 
elements as R^. Then the jth main diagonal elem 
сї (fj = 1,2, +++, n): 


sponding main diagonals 
e and also later. Let 87° 
the same main diagonal 
ent of 5° itself is simply 


S? = [et 03, +++ Let]. (7) 
If R is nonsingular, there exists a nonsingular matrix F such that 
В = FP’, (8) 
Ё сап Бе сһоѕеп іп infinitely many ways for (8) when n 2 2, but always we 
can rearrange variables to find an F of the form 
pel4 0 | (9) 
B. g 


submatrix of order т, B is of order 
and of order n — m. From (8) and (9), 
AA' AB' 

BA’ BB’ + cc 
It is easily verified that the inverse of F is giv 


where A is a nonsingular square 
(n — m) x m, and C is nonsingular 


"| 


: (10) 


en by 
= Ae 
pA 0 (11) 
-С7ВА (+ 
From (8), g^! = (PYF, or using (11) 
d G н 
ЕС 
Н (соу 
where 
@ = (AA) 4 (A)B'(Cery?p 4-1 (18) 
and 
H = —(CC7)? B4. (14) 
Now consider th. 


€ special case where (07 18 à diagonal matrix. Then 
(CC")! is also diagonal. According to 


(12) and (7), (CC) constitutes the 


| 
| 
| 
| 
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lower right-hand submatrix of S^, or CC’ constitutes the corresponding 
oe of S and defines the ož forj = m + 1,m--2,:-:,"n. If we subtract 
E: is [iesus CC' from the lower right-hand corner of R in (10), we are 

early left with a reduced R that is Gramian and of rank m, it being the 
product of [A B]' and its transpose. Thus we have: i 


Ра ути 1. If R can be factored into an F of the form (9) where CC' is 

co e £u А and C are nonsingular), then the main diagonal elements of 

an a eg cifoj-m-lim-2 n" If these n — m o; in 

B su irae from the corresponding main diagonal elements of R, the 
g matrix will be Gramian and of rank m. 


eu Кы to Theorem 1, when m is the actual minimal rank possible 
ES mias R — 07°, then the first m diagonal elements of U? can be set 
E es о zero, and Ше last? — m diagonal elements equal to the corresponding 
nn Biven by CC’. Thus, the last n — m of the оў serve exactly as rank- 
imizing uniquenesses, or the equality in (4) holds for j = т + 1, m +2, 
at yii 
Toti x 
- d that the first m uniquenesses implied by Theorem 1 are zero and 
"ied s the c? . If the first m оў were also subtracted out from the main 
al of R, then the resulting R — S? would in general not be Gramian, 
or of minimum rank (cf. 4). 
с 1 holds even when the m in it is not minimal. It is always 
Uy е to use the Theorem for the case where C is of order 1, and hence 
is necessarily à diagonal matrix. This provides: 
is subtracted from the 


Corollary. For any nonsingular R, if any one ei 
Iting matrix is of rank 


co ; E 
а “ee main diagonal element of R, then the тезш 
ne in his discussion of the 
known to mathematicians 
gonalization), but without 


“ iem uis ^d partly indicated by Thursto 
as the on od of matrix factoring (better 
аваар a 1 а ог Gram-Schmidt process of ortho, 
We үнү that his implied uniqueness was exactly e; (8, р. 308). 
many of the QUS completed showing that there are many matrices for which 
АЫ sicions M ei serve as rank-minimizing uniquenesses. Also, we have 
а, Gramia; oon that any one of the c; alone will reduce nonsingular R to 
n matrix of rank n — 1. 


We 1 V. Equality in the Limit as n 9? 
eh j Р 
really to е seen in Part III how, if а *unique"-factor variable is 
asn— = сс determined for a given u4 , then we must have 0/03 > 1 
size of m dior отав to (5) and (6). This conclusion does not depend on the 
У in particular оп whether m remains finite or becomes infinite 


as n : 
— o. It thus applies to ordered factor theories—such аз the radex, 
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with its simplexes and circumplexes (5)—as well as to limited common-factor 
theories like those of Spearman ‘and Thurstone, whenever the ó-law of 
deviation (5, p. 308) holds for the unique-factor variables. У 
Thus, a general sufficient condition for c? to tend to uj when uj > 0 is 
that r; > 1 or r* — 1 as n — œ. This holds for each j separately. 
A less general sufficient condition, and one that does not necessarily 
hold for any one j but only for “almost all” j, is given in 


Theorem 2. If R is nonsingular for all n, and if lim,.. m/n = 0, then 


n 2 
limi ууш = 
For all except possibly a zero proportion of the J it must be that lim... u2/o2 = 1. 


'The condition that m/n — 0 holds in partieular for the Spearman- 
Thurstone approach to factor analysis, which postulates that the number of 


common factors should be small compared to the number of observed 
variables. 


Since 12/02 < 1 for all j, according to (4), 


we must have the mean 

ratio also bounded above by unity: 
lau х 
n $i x (18) 


> 0 for all j and n, 
first conclusion of 


As is well known, if R — U? is Gramian and of rank m, we can write 
Ri= AA’ + Us. 

Where A is some matrix of order n X m and of c 
the symmetrie matrix of order m: 


Q-I,- ACUSA 
where J,, is the unit matrix of order т. It has 
Q is Gramian and nonsingular, and furthermore 

a=. 0208 U^ AQ" Arp? 
It is easily verified further, from (18) and (17), that 
АВА =T, — ORA 


been shown in (2, 92) that 
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Since the left member of (19) is clearly Gramian, so must the right member 
be. Indeed, it is known that Im — 07: is the covariance matrix of the predicted 
values (from the observed n variables) of any т orthogonal common-factor 
scores underlying loading matrix A (6). Let q* denote the kth main diagonal 
element of Q^, or the variance of estimate of the kth common factor, and let 
p; be defined as 


p=1- g" (Е = 1,2, ~" ‚ т). (20) 
Then p; is the square of the multiple-correlation coefficient of the kth common 
factor from the n observed variables, and 

O<p<i1 (k = 1,2, °°» т). (21) 


Therefore, the frace—or sum of the main diagonal elements—of In — Q^ 
satisfies 


(Un -@) = Èp т. (22) 


We are particularly interested in the trace of UR, for clearly—re- 
membering (7)— 
$ n 2 
tr (UR) = (US) = X (23) 
1=1 OF 
Since the trace of a product is unchanged if order of multiplication is reversed, 
(АЕА) = tr (AA'R™) = tr Un — UR), (24) 


the last member following from the middle member by recalling (16). There- 
fore, taking traces of both members of (19) and using (23), (24); and (22), 


2a Sane i: (25) 
kel 


SER 
S45 


b 
1u aby nae (26) 
1252 =1 2 Pe A 


Clearly, if m/n — 0 in the last member of (26), the middle members must 
tend to unity, or Theorem 2 is established. E 
Notice that Theorem 2 could be rephrased to say that almost ай Tu 1 
or almost all unique-factors must be determinate in the limit. It is interesting 
to see this in a slightly different way. From (5) and the middle members of 


(26), 


Lig у Уур (27) 
о hg ei" 1, 
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or 


х May (28) 
Vp ғ 


P tend to unity as n — о, it must be that 
m/n — 0. This does not require m to remain finite, of course, but only to 
increase at a less rapid pace than does n. 


VI. Increase in Т, nformation with т, 
A desirable property of estimates of communalities is that they should 
improve in general as n increases, An: 


factor analysis are usually regarded 


niverse of variables (n = 


the increased informa 

Similarly, if the 
universe of Observed 
this again makes о? 
creases, 


jth unique-factor Scores are defined uniquely as for the 
variables, rj must in general increase with n. From (5), 
an increasingly better estimate of the fixed uj as n in- 
"Thus, the ] 
new information is br 
sumptions at all. In b 


mation can make p? — рз fi $ , this new infor- 
Pi — h; for all or almost all j. 


ot 
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With any nonsingular correlation matrix R is associated another non- 
singular correlation matrix R* defined by 


R* = SRS. (29) 


R* is clearly Gramian, for R^ is Gramian and S is a diagonal matrix. The 
main diagonal elements of R* are all unity from the definition of S and the 
fact that 1/02 is the jth diagonal element of R™. Indeed, R* is the correlation 
matrix of the anti-images of the n variables of R (cf. 3, p. 294f). Regardless 
of the statistical meaning of R*, it is a perfectly good correlation matrix when 
т is finite, and we can seek a diagonal Gramian matrix U* that will leave 


. R* — U* Gramian and with minimum rank m*. This will lead to the interest- 


ing and important inequality for the case where no c is a uniqueness nor 


equals unity: 
m -rFm*zn (ut < x1j-2152 7,9 (30) 


The restrictions that S? — U? and I =S pe nonsingular are essential here 
(consider the counter-example where S — R = R* = D). That o z£ 1 (I — S 
be nonsingular) implies that each variable in R has at least one nonzero 
correlation with some other variable. 
: According to (30), if m/n is small, then m*/n must be large. Conversely, 
if m*/n is small, m/n must be large. This is rather paradoxical in view of the 
fact that R* can always be reduced to rank m by subtracting out the diagonal 
matrix SUS (= 807°). This follows by pre- and post-multiplying (18) 
through by S, remembering (29), and noting that the second term on the 
right is of rank m. Conversely, R can always be reduced to rank m* by sub- 
tracting out S* 0%7°, where S* is the diagonal matrix defined by the main 
diagonal of R*-*. Thus, if all diagonal-free submatrices of R have rank less 
than n/2, so must those of R*, and conversely. Regardless, (30) holds. 
In effect, then, (30) implies that to every R for which оў # uj or 1 for 

au j and where m < n/2, there corresponds an R* which is a generalized 

Heywood” case (cf. 4, 159f). Although all diagonal-free matrices have 
small rank in R*, no communalities can be found to make Ё — 0° of equally 
small rank and yet be Gramian. It must be that m* z n — m. This again 
emphasizes that the case m < n/2 may be the exception, rather than the 
rule, for correlation matrices. And it is interesting that this paradox arises 
precisely for those cases where no c; equals the corresponding ш. i А 
. To establish (30), we first recall the theorem (4, 1575) that if S — U 
is nonsingular, and if s is the non-negative index of R — S^, then 


tam) (59-9. (81) 


Now, the proof of (31) in (4) can be modified to take care of the case where 
5 = U is possibly singular, to establish the weaker but more universal 
inequality р < т, where р is the positive index of — 8°. We shall not take 


° 
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space to prove this modification here, 


but shall merely state it in terms of 
our needs for Ё*: 


p* S m*, (32) 
where p* is the positive index of R* — 
S*' — U* to be nonsingular, 

Now, from (29), +: = SRS! 
of R are all unity, 


S*, and we do not necessarily assume 
s Ог since the main diagonal elements 
Se -g. (33) 


It is interesting to note that (33) and (29) imply that (R*)* = R, or R is 
to R* as R* is to R. 

Statistically, (33) implies that the relative predictability of the jth anti- 
image from the n — 1 ini ii i 


\ g original variables. From (29) and (33) 
we can write the identity 
¥ 


N RE S? = Sm DS. 
Sylvester’s “law of inertia” (cf. 4 


p* equals the positive index of pt 
number of late 


ity. Hence p* ес uals the number 
of latent roots of R itself which are less than uni e 


stp tan QE = |»), 
Inequality (30) follows from (81), (32), апа (85). 


"best possible" inequality; we must show that 

holds. Tt suffices to consider an 
› Say M > 1 with multiplicity f 
must be that 


(35) 


x Mf, йш ge (36) 
M. : ; 
У 2 Ai Fra sare pe i (4, 159), and hence m = f by considering 
tu Ny ашап and of rank f: m* — fx b i 
on £^. Since f + Р = n, (36 ides i, y analogous reasoning 
in (00) fins » (36) provides а Special 


case where the equality 
Inequality (31) by itself is similar t possible” Consider the 
case where R* has two distinct latent roots, say А, < | Жа lil ^ fa 
and M < 1 with multiplicity P =n — p*. Since рж-1 _ РА = BRS" = 
|. im S )S^', p is the positive index of R — g? while p* is that of 

- Also, since no root vanishes, p = s or the positive and non-negative 


ly a “bes 


Ld 
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indices coincide. Since Ra ШТ = S"RS^ — М is Gramian and of 
rank p = s, so must R — Az S? be, or the equality in (31) must hold for this 


case. 


VIII. Relation to Image Analysis 


| The ratio of wj to оў indieates the relative predictability of the jth 
unique-factor scores from the n observed variables of R, according to (5). 
Closely related is another parameter developed in image theory and denoted 
by à; , namely, the variance of the difference between the respective scores on 
the jth anti-image and the jth unique factor. 1% turns out (3, 298) that ôi 
can be computed as the simple difference 

0 = 0-0. (52:729 (87) 
Непсе, а necessary and sufficient condition that gia asm is that 
8; — 0. This implies that the unique-factor scores must be essentially the 
total anti-image scores from the universe of content. Here we have the 
individual anti-images themselves as increasingly Fi ier estimates of the 
unique-factor scores as n > ©. This problem of estimating scores is perhaps 
even, amore basie than that of estimating only over-all parameters, such as 
uniquenesses, which are based on the scores. Estimating PM» 5° has the 
important property of tying in directly with the score estifmation problem 
via image analysis. 
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RANK-BISERIAL CORRELATION 


Epwarp Е. Советом 


UNIVERSITY OF TENNESSEE 


" A formula is developed for the correlation between а ranking (possibly 
including ties) and a dichotomy, with limits which are always 1. s 
formula is shown to be equivalent both to Kendall's 7 and Spearman's p. 


Suppose we have two correlated variables, one represented by а ranking 
(possibly ineluding ties) and the other by a dichotomy. The dichotomy 
may be considered a ranking concentrated into two multiple ties; its ties, 
however, do not represent equal measurements (or judgments of equality) 
on а continuous (or at least a many-step) variable. Rather, the ties represent 
a broad grouping of the data into two categories, or possibly an actual 
two-point distribution (sex, e.g.). Since the number of distinct ranks in the 
ranked variable will always be much greater than 2 and will equal N in the 
untied case, exact rank agreement of the two variables, pair by pair for each 
individual, is impossible. In this situation we desire a. coefficient, which will 
still have attainable limits +1 in all circumstances. It should be +1 when 
all ranks in the “higher” category of the dichotomy exceed all ranks in the 
“lower” category, and —1 when all ranks in the “lower” category exce e 
all ranks in the “higher” category. It should be strictly non-parametric, 1.8., 
defined wholly in terms of inversions and agreements between pairs of rank- 
pairs, without use of such concepts as mean, variance, covariance, Or Te- 
gression. Finally, it should resemble the usual rank correlation coefficients 


in some reasonable sense. 
Let R. represent the dichotomy, with categories Ё. + em us 
let E, represent the ranked variable. Ties in Ё, are toaa ie 
rly as possible 


mid-rank method. We then arrange the ranks R, in as nea ose 
“high” and rank 1 “low, 


the natural order (N, N —1 i EN 

—1, --. , 1), with rank à 
and allocate them to the MM р. + and Е. — as in the following 
example: 
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R: + R.— Inv. Agr. 
9.5 : 
9.5 
8 4 
6.5 3 
6.5 9 
4.5 3 (D 
———————— 
4.5 3 
2.5 
2.5 
1 _ 
№=6 N,=4 Q-2 P=% 


= —, the relation 
locations is immaterial. Thus, in (1), the first 


we should find that P= 
result merely from an in 
R, —. Our coefficient m. 


"ав = (P — Q)/P 


— (2) 
if Q = Оаар =P 
Q. 


a —lifP =OandQ = 0 


It will be 4-1 
and 0 if P = 


-— — 


that in the Situation in which the 
greement; 


a 5 for every number under R. +, 
; however, so f 


ar passed over, in which Pmax 
iN, . This case is illustrated in our example. If we 
ation for P — Prax With these data, we have: 


2 
is one case 
cannot be as great as №, 


set up explicitly the situ 
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Р. + R.— Inv. Agr. 
9.5 " 
9.5 2 
8 4 
6.5 : 
6.5 M 
4.5 3 

(8) 
4.5 
2.5 
2.5 
== 1 ES 
N,=6 №, = 4 0 = 0 Р = 28 
N,N. = 24 


(0) Р 
ne agreement is lost because the lowest rank under Ё. + is tied with the 
ight be a triple or multiple tie 


meg under R, —. In other cases there mi 
Por Du а dichotomy. We shall term а tie at this point a bracket tie. 
Ron grab 89 et tie, the value of Poss will be reduced from N,N. by unity 
бачу лр of members of this tie one of which is under R. + and the 
in the memes Ж —, after Fy has been rearranged to be as nearly as possible 
ену ^ order and allocation under E. + and R, — is made in such & 
under В S to preserve the original values of Ni and N: . If f, 1s the number 

. + participating in the bracket tie, and f, the number under Ё, —; 


p ЖИГ 
max = N,N, — А , and our formula becomes 


_ Lupi 
Tes = N,N, — hile (4) 


Physically, it is not necessary to rearrange the original data in order 
columns E. + and 


to 
R dr tit, . We merely draw à horizontal line across ¢ 
z — in (1), at a level which leaves N, cases above the line and N: below it. 


Si Si : 
ince the original arrangement in (1) was with R, in as nearly as possible 
any group of identical 


а order, а bracket tie will then consist of lenti 
Th ers, some immediately above and some immediately below this line. 
e number above is ћ and the num For the example of (1), 
We find by (4): 


ber below is £s - 


21— 2 
a anni rt .826 
Ta = (gy4) — (00) 
of Clearly Tre is а Kendall-type coefficient, since Q and P are the numbers 
S unweighted inversions and agreements, respectively (2). But it is also & 
pearman-type coefficient. Durbin and Stuart (1) have shown that, in the 
by (U — V)/(U — V) max › where 


untie: aan eae: 
d case, Spearman’s coefficient is given 
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eight may come from either Ё, or Ra a 

x for the cases corresponding to Kendall's 
1 able to prove in these cases that the 
max Are necessarily equal in general to 


mulas based on Zd", but he has verified 
each of them on several sets of numerical data. 


=» all weights will be equal (and equal to 
and it follows at once that 


differs only by chance from Pre = 0 may be 


extension of the Wilcoxon test (3). 
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A NOMOGRAM FOR FACTOR ANALYSTS* 
Р. D. DEUVEL 


UNIVERSITY OF CALIFORNIA 


When a new reference vector is chosen graphically from the plane of 
two old ones, its direction cosines as well as the projections of the tests on 
it are most easily computed by applying certain multipliers d and Sd to 
quantities which are already known. The nomogram quickly supplies d, 
after S has been read from the graph. 


The nomogram accompanying this article reduces the computing work 
in what is perhaps the most popular of the graphical rotation methods of 
factor analysis, namely, the diagrammatie method explained in Thurstone 
(1, pp. 194-216) with rotations made in one plane at a time. Use of the 
nomogram will be explained in terms of the following figure: 


S=+.647 
Cos=+.383 


IS + Cos] = 1.03 


As usual, A and B represent trial reference vectors and the dots represent 
the projections of the test vector termini in the plane determined by А and 
B. (Following psychological usage, we refer to the “given” or analyzed 
variables as tests.) The cosine, 383, of the angle 0 between А and B has 


been recorded in the lower left-hand portion of the diagram; it was obtained 
*Suggestions by Norman Li ichols, and Karyl Atherton have been 
incorporated in the nomogram. ME лоше А the necessary computations. 


In addition, I am obligated to Katherine Eardley, scientific illustrator, for her care in. 
lettering and inking the original. 
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in verifying the linear independence of А and B. The figure suggests re- 
„placement of A by a new trial reference vector A”, collinear with the dashed- 
line vector A’, for five tests have nearly vanishing projections on A’. From 
the diagram we read the vector equation А' = A + .647B. The projections 
of all the tests on A", as well as the direction cosines of A", are required. 
More generally, the method under discussion will always yield a “long 
reference vector" А” related to known reference vectors A and B by one of 
the equations A’ = A + SBor —A’ = A + SB, for some number S between 
—1 and 1. When A’ = A + SB, the inner product (V, A") between any 
vector V and the unit vector A” collinear with A’ is 


(V, А”) = 7, A) + SAV, B), Q) 


where d is (A’, А”), the reciprocal of the length of A'. If V is a test vector, 
(1) gives the desired projection (V, A") of V on A” in terms of the known 
projections (V, A) and (V, B) of V on A and B. Similarly, if V is one of the 
orthogonal basie vectors, (1) gives the desired direction cosine (V, А”) of 
A" with respect to V in terms of the given direction cosines (V, А) and 
(V, B) of A and B with respect to V. Thus, when d is known, all the required 
quantities are obtained from (1) by applying the multipliers d and Sd to 
columns containing the relevant inner products (V, A) and (V, B). The 
SUA —A' = A + SB is accommodated by simply negating the right side 
of (1). 

_ , Upon noting that (А, B) is the cosine of the angle @ between A and B, 
it is seen that 


d = (| S + cos 0 |? — | cos 8 ае) (2) 


а graphable function of | S + cos 0 | and | cos 0 |, and in fact, the function 
represented in the nomogram. To determine d, therefore, it is only necessary 
to perform the addition S + cos б, enter the nomogram with arguments 
| S + cos 0 | and | cos @ |, and read d from the appropriate inside scale. To 
read the nomogram: 


1. Locate | S + cos 6 | on left-most stem and | cos 8 | on right-most stem. 

2. Align a straightedge through the two points thus found. 

3. Pick out appropriate inside scale. This is the scale whose label (at 
top) combines the labels of the scales on which | S + cos 0 | and 
| eos 0 | were found. 

4. Read d where straightedge crosses appropriate inside scale. 


Generally, S is read from a graph and cannot be identified with better 
than two-place accuracy. In such cases a gratuitous third digit may be 
appended to S, so chosen as to render the third digit of the sum S + cos 0 
zero. This device makes for greater ease and precision in locating values of 
| S + cos 0 | near the top of the left-hand stem. 
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In our example, the value +.64 - for S was read from the graph, and 
the final digit 7 was selected to complement the third digit 3 of cos 0. The 
addition S + cos 0 = 1.03 was performed directly on the diagram. The 
value 1.03 for | S + соз 0 | was then found on the B-scale of the nomogram; 


the value .383 for | cos 0 | was located on the X-scale; hence, d — .723 was 
read from the B X-scale. 


The scale factor of the nomo; 
of the BX-scale to the upper ге, 
been designed, how 


analyses, indicate t 
tations of the above type, the liberal A X-scale 


r t i elfare, University of California, Berkeley 
4, California. The copies have stems approximately 92/7 
on 82" x 11” index-card paper. 
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A NOTE ON 
THE ESTIMATION OF NONSPURIOUS CORRELATIONS 


"WiLLIAM Н. ANGOFF 
EDUCATIONAL TESTING SERVICE 


A method is provided for estimating the nonspurious correlation of 
a part of a test with the total test. Two cases are considered: one in whic! 
the actual subtest is parallel to the total test, the other in which the actual 
subtest is not parallel to the total test. 


A problem that frequently arises in the examination of test data is that 
of estimating the degree of relationship of a test with a subsection of items 
drawn from the test. In general two methods are available: one is to correlate 


the subtest with the total parent test; the other is to correlate the subtest 
with the complementary subtest resulting from the subtraction of the first 
subtest from the total. In the first method, it is clear that spuriousness 


exists; even if the subtest is totally unreliable and correlates zero with the 
Jation of the part with the total 


complementary subtest, the computed corre 
and roughly in the proportion 


would result in a value greater than zero, 
represented by the subtest in the total. In general the correlation would be 


ти = Gi + тов) / 00 (1) 
where 2 is the total test, and J and h are the complementary subsections of 
the total test. Even if ria = 0› "i is still greater than zero, merely by virtue 
of the presence of j in the total test. | ў 

The second method, on the other hand, defeats its own purpose—it 
yields a correlation of the subsection j with the complementary subsection 
h, not a correlation with a test of the length t. s 

In order to estimate the nonspurious correlation, вау Tiu consider test 
1 to be an unspeeded test of power, and, as before, to comprise two ake” 
jand h. Also consider 2 hypothetical test, j^; exactly parallel vus of kie sg 
effective length (1) to J- Subtests j and № need not be parallel forms. 

(2) 


Tj - (55°95 + ryenon)/ 0 . 
e j and j' аге parallel, 
xpresses the reliability of test J 


as the correlation between para i 
i i tten 

E i А written aS Tih and ту, Will be wri | 

Pai ce к=» ДЕ] correlation, will 


аз r;; . However, the notation Tj^ › 
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be retained to distinguish it from the spurious correlation, r;, . With the fore- 
going modifieations in notation the formula for nonspurious correlation is 


Tja = (rio; + туһоһ)/т‹. (3) 


In the actual situation, the reliability, r;; , will probably be estimated best 
by one of the internal consistency reliability formulas appropriate to power 
tests. 

If now a further restriction is placed on test /, namely, that j and h be 


parallel (but not necessarily of equivalent length), then further simplification 
is possible. It has been observed (1) that under this restriction 


Typ Tui ; i (4) 
and also 
Тт, сач. 
та = 6—1. 
E Tilo, — Tj,0;) н 
Substituting (5) in (4), 
ruri, — о) 
uiri er тет ESSERE 
It may also be seen that if ro c; i 1 1 
48: — c; is substit i i 
Ting, [see equation (1)], then g Du н 
н Tn = (тне, + тиса — o) /o, . (7) 
Finally, substituting (6) in (7), 
Тис, — о; 
Tja = 1i. 
UT m тне, £ 


iD 
m i^ e p thus presented for estimating the nonspurious corre- 
(3) and in its EM (m иа Se eet eam. V 
drawn from the total test, Т} cse equation is imposed on the kinds of items 


х hese е і 
estimated correlation of а sub DR RUE 


plifications may be made in (8). From (1) it 


is seen that the numerator of (8) may be written Tinon . It may also be seen 


=æ 
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that the denominator of (8) may be written Taan since 


te = Tan = i — Tj0;)/95 - (9) 
Thus (8) may be written 
Tja = Tinn Tun = Tia Thi. - (10) 
Also, from (6) and (8) 
тө = 711 э (11) 
and finally, from (5) and (8) 
fpa mls (12) 
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THE VARIANCE OF THE NUMBER OF MUTUAL CHOICES IN 
SOCIOMETRY 


Leo Katz 
MICHIGAN STATE UNIVERSITY 


AND 
Tuurtow В. WiLsON 
UNIVERSITY OF NEW МЕХІСО* 


The variance of the number of mutual dyads in а sociometric situation 
where each member of a group chooses independently and at random is 
derived for unrestricted numbers of choices per group member, as well as 
fora Ne number of choices. The distribution of the number of mutuals is 
considered. 


I. Introduction 


In a sociometrie test, a number of the pairs of group members (dyads) 
may show mutuality. A mutual dyad is defined by the two subjects selecting 
one another. When each member of the group makes a fixed number of 
choices, the number of mutual dyads has been assumed to have a binomial 
distribution with p, the probability of a success, equal to the probability 
that a given dyad is mutual, and n, the number of binomial observations, 
equal to the number of dyads (1, 7). Several writers have pointed out that 
since sampling is without replacement, this is not correct (2, 4, 6). Let M 
represent the number of mutual dyads for а group of N numbers. If each 
member independently makes d selections at random from the N — 1 other 
members, the probability that a given dyad is mutual is @/(N — 1)’, and 
the number of dyads is N(N — 1)/2. The expected value of M is not affected 
by the non-replacement in sampling and is given by 

Nd* 
E(M) = np = aN = 1) а) 


We shall show that for а fixed number of choices 
2 


Var (M) = zan(ı - s. 


ate to the binomial distribution, 


(2) 


rather than the variance appropri: 


d 2 
Var (binomial) = npg = во]. = G4 a) l (3) 
*We are indebted to Robert Bush and Hartley Rodgers of Harvard University for 
helpful criticisms. 
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The ratio of the binomial estimate to the variance of (2) is (М — 1 + d) AN = 
1 — d); the binomial formula will appreciably overestimate the variance of 
М if the size of the group is small. 

When the members of the group do not make the same number of 
choices, it has been assumed that one may approximate the variance of M 
by using the average number of choices, d, in the formula for fixed d. We shall 
develop the expression for the variance of M for the general case of unrestricted 
numbers of choices and examine the approximation with d. Finally we con- 
sider the distribution of the number of mutual dyads. 


П. Variance with Fixed Number of Choices 
We first derive the variance of M for the case w 
makes the same number of choices, d. Define a random variable X;; for the 


particular dyad composed of individuals i and j so that X;; is 1 or 0 according 
as dyad 7 is or is not mutual. X;; is a binomial variable; thus 


Var (X,) = d’ (1 Ф ) 


waa wo " 
Since 
Mice OS xs (5) 
where the sum is taken over all distinct dyads, we have 
Var (M) = 5 Var (X;;) +2 px Cov (X 5X4). (6) 


The covariances are summed 


ae rlance of such a pair is zero. A pair of 
yads may not have two members in common, but a pair may have one 


B in common. There are NW — 1)\(N — 2)/2 such pairs. The expected 
value of the product X i7 X; is just the probability that both are equal to 1: 


З, 
Вх) С СЕЕ 
(Xa; D OUS: (7) 
where dyads ij and ik have one member i i 

ne oup a mp er in common. We find the covariance 

3 

Соу (ХХ) = -ENW —1— d) 

ло Ҹ 


Substituting the values from (4) and (8) in (6) an 


the variance of mutual choices given by (2). 


ПІ. Variance with Unrestricted Numbers of Choices 


We now derive the variance for the case 
C wher 
unrestricted numbers of choices, Let d, d NEU Dan E 


enote the number of choices actually 


here each member 


d simplifying we obtain 


J pc ———————— 
кн = == = E 


— a —— RR 
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made by subject i. The probability that dyad 1] is mutual is given by 


р = Pr (Xi; = 1) = == (9) 


The probability that both dyad ij and kl are mutual is 
d; d; d; di 


риљ = Pr (Xi;-Xu = 1 = wD (10a) 
if all four individuals are different, while 
Dive = Pr (Х.Х = 1) = cu аа (10b) 


N—1(N-2' 


if the two dyads consist of three different members. 
Conventional methods for obtaining the variance (5, pp. 60 ff.) involve 
computing two sums, 


8, = Da, (11а) 
8 = Уу Pum › (11b) 


where both sums are evaluated over all distinct sets of subscripts. Considering 
(5), (9), and (11a) we see that the mean of the number of mutuals is S; . 
'To express Var (M) as a function of S; and S; we first note that 


Var (Xa) = pii — (рг), (12) 
апа 
Соу (Х,;;Хы) = ргы — Puta - (13) 
Summing over the variances and covariances yields 3 
> Var (Х.) = Si — DE (p^; (14) 
and 
У) Cov (XXn) = 8 — P spa. (15) 
Since 
S? = У) (р) + 2 E popas (16) 


substitution of (14) and (15) in (6) produces, after simplification, 
Var (M) = Sı + 282 — Si. 


We turn next to the computation of S; and Ss 
definition 


(17) 
. The value of 8; is by 


¿<i 


eet SS РЗ 18 
8. = aon 25 6 (18) 


302 PSYCHOMETRIKA 


It is, of course, necessary to require 7 < j in order to prevent duplication of 
cases in the sum. The summation appearing in (18) is the second elementary 
sum of the numbers d; , usually written 


а = > dd). (19) 
i<j 
These are related to the more familiar power sums, 
„= У а", (20) 
by the relations (3) 
lx 
а = 5 (81 — 8), (21a) 
2lqg 
аз = 31 (si — 38,8, + 2s), (21b) 
а, = EN. 4 6 2 2 
a git — бё + 882 + 855, — 65). (219) 


"Those shown will suffice for the present com 


i putations. I ti 5 А 
established from (18), (19), (20), and (21a) th ns. In particular, we have 


at the mean number of mutuals is 


; 1 2 
EM) = gay ipe - xn. en 


If all d; equal d we obtain (1) above. 


The value of S; is somewhat mor 
of four diff 


have 


3 
mr ўў 27 1444,4, 


i<icke 
М i (23) 
r A ыы. 
(№ — DN — 2) У, PLACA — 1) d; d]. 
Making use of (21а) and (21b) we have, after some reduction. 
? 
1 
„= aN — DF (st — 6sis, + 382 + 85:5; — бз) 
(24) 


— — — 
2(N — D'U 2) e — S2 — THREE — si да — ш. 


S 
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Combining this with previous computations, we obtain 


Var (M) — D (—2sis, + 52 + 48:8: — 38) 


А уз. ж E E dd pee " 
T (N — D*N — 2) (sis2 S2 25,8; + 2s, si + 35152 253) (25) 


se Еси ($i — sj). 
When all d; equal d, (25) reduces to (2). 

As examples with unequal numbers of choices, consider two sociometric 
measurements on 10 individuals. In the first case, two subjects choose two 
persons each, five choose three, and three choose four. The power sums are 
31, 101, 343, and 1205 for sı , S2 , % , and з, respectively. Equations (22) 
and (25) give E(M) equal to 5.31 and Var (M ) equal to 2.30. Observe that 
the use of d equal to 3.1 in (1) and (2) would give the very close approxi- 
mations, Z(M) equal to 5.34 and Var (M) equal to 2.29. 'This happy situation 
would not obtain if the numbers of choices were considerably more variable, 
аз in the second example; five make a single choice and five make seven 
choices. The power sums are 40, 250, 1720, and 12,010. From (22) and (25) 
E(M) is 833 and Var (M) is 2.33. With d of 4, we obtain the values 8.89 
for E(M) and 2.74 for Var (M). 


IV. Distribution of the Number of Mutual Dyads. 


In principle, we could determine the distribution of M exactly by con- 
ventional methods (5, p. 64). Denote the maximum possible number of 
mutual dyads for a group by Max. We define S, as suggested by (11а) and 
(11b) above. Then 


Er = m= Sa oe в, + (* Ф Pa 


m (26) 
eee Cs 
We have used (26) only for the case where each person makes а single choice, 
where 
( күөн — B i (2) 
_ Халы Bk 2 4 2, (27) 


m= mXN — 17 


and ls N/2. 
nd Max equals №/ oups with roughly equal d, M has an 


We conjecture that for large gr | : 
Approximately Poisson distribution; from (2) one readily notes that with 
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increasing group size and r held constant, the variance of M approaches 
E(M) and the covariance term approaches zero. 

The senior author is currently engaged in determining higher moments 
for various common fixed d by methods similar to those used to produce the 


variance for the general case. These higher moments will permit determina- 
tion of the distribution of M for small groups. 
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A NOTE ON JENKINS' 
"IMPROVED METHOD FOR TETRACHORIC r” 


Јоѕнол А. FISHMAN 


COLLEGE ENTRANCE EXAMINATION BOARD 


Some readers who will be delighted to utilize Jenkins’ method and 
tables for estimating the tetrachoric correlation (1) may be puzzled to dis- 
cover that no explicit provision for negative correlations is included. If we 
follow Jenkins’ instruction to "letter the fourfold table so that a is smaller 
than d and ad greater than bc" four possible arrangements may obtain: 


c|d bla a |b d|c 


a| b dic cid bla 


Of the above four arrangements, the first (which is the one illustrated by 
Jenkins) and the second are indicative of a positive correlation and the third 
and fourth are indicative of a negative correlation. If either of the latter two 
arrangements does obtain, then the final correction {obtained by multiplying 
the base correction by the multiplier, as in step 5, p. 257, (1)] should be 
(algebraically) added to, rather than subtracted from the negative un- 
corrected т to obtain the corrected tetrachorie т, which should, of course, 
be given a negative sign. The important fact to keep in mind is that. the 
correction always reduces the absolute size of the uncorrected tetrachoric т. 

A few words also might be in order concerning the location of decimal 
points in Tables 2 (Base Correction) and 3 (Multipliers for Base Correction). 
Whereas in the former table the omitted decimal points consistently belong 
before the first digit of the reported three-digit table entries, in the latter 
table the omitted decimal points belong before the first digit of two-digit 
table entries, between the first and ‘second digit of three-digit table er 
and followed by a zero for one-digit table entries. Thus, whereas ae е 
entry of 106 should be understood as .106 in Table 2, it should be {а T 8 
1.06 in Table 3. Furthermore, 90 is .90 (as illustrated by Jenkins, p. 257) 
but 9 is .09 in Table 3. 


1. Jenkins, W. L. An improved method for tetachoric 7. Psychometrika, 1955, 20, 253-258. 
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BOOK REVIEW 


Mathematical Models of Human Behavior. Proceedings of а Symposium Sponsored by 
Dunlap and Associates, Inc., and the Commission on Accidental Trauma, Armed 
Forces Epidemiological Board. 1955. vii 4- 103 pp. 


Those who expect to read this book in its entirety, or nearly so, would do well to 
turn first to Professor Lazarsfeld’s Concluding Remarks, since they provide some degree 
x unification for what is otherwise a rather disjointed collection of reports on several 

iverse lines of investigation. Those who do not expect to read the entire book would do 
well to read at least Professor Lazarsfeld's Concluding Remarks for a brief but lucid 
с the function of models, especially in the behavioral sciences. He distinguishes 
ei d ae and dynamic models, makes passing reference to the predictive function 
tis "X s, ап concentrates on their linguistic function. The linguistic function he divides 
ree parts: organizing, analytical, and mediating. Naturally his remarks refer to 

the papers in the Symposium, but they are quite intelligible in themselves. 
Бы The Symposium itself was held in February of 1954, in connection with a study 
eing made by Dunlap and Associates, Inc., on the application of mathematical techniques 
to the study of accidents. Since accidents are “partly the result of human behavior,” and 
since many experts were already engaged in devising and studying mathematical models 
of human behavior, the proposal was made to invite some of these experts to meet together 
to describe their work and to participate in informal discussions. This publication contains 
the papers, but unfortunately not the discussion. 

Of the ten papers included (not counting the Concluding Remarks, and not counting 
2 paper by Lorge and Solomon to be published elsewhere) only one deals with accidents. 
This is one by H. H. Jacobs, who discusses the difficulties in trying to separate the effects 
of contagion from individual differences as to liability. Two of the speakers, Bush and 
Estes, discussed somewhat related stochastic learning models. The other seven papers 
had to do, more or less directly, with utility, or decision making, or both. 

For general background on models, the paper by Coombs and Kao might be classed 
along with Lazarsfeld’s Concluding Remarks. This is actually the first section of a report 
on multidimensional analysis. and one feels suspended in mid-air at the end. As an in- 
ducement to learn more, this paper succeeds very well. At the other extreme, a paper by 
Luce on the formation of coalitions in game theory is largely for the experts. The other 
five papers are more self-contained, and more directly concerned with utility and decision 
as such. 

Professor Lazarsfeld gently chides the speaker: 
although, as one of them remarks, the gambling situatio ; 
realistie contact with the individual's utility function. Merrill Flood's little “Group 
Preference Experiment” does not make use of a conventional gambling situation, but it 
does afford, to each of the subjects, а possibility, without certainty, of some gain. A collec a 
tion of objects is shown to a group of individuals, and certain broad conditions are laid 
down according to which one of these objects can be had for the group to dispose of. They 
are left to decide which object it shall be and how it shall be disposed of. Disposition might 
be by lot to one of the group, or by auction or sale with proceeds divided, and other possi- 


bilities can be conceived. . 
uals, pointing out that even when the 


Marschak is concerned with decisions by individ Я Q 
individual may make different choices 


outcome of an act is known with certainty the same 1n U А i 
at different times. He considers various hypotheses that might be made in formulating а 
model of such inconsistent behavior. Markowitz discusses a hypothesis of Friedman and 
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s for their preoccupation with gambling, 
n provides the most direct, and 
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Savage for explaining insurance and lotteries; Markowitz replaces it by one of his own 
which seems to accord better with well-known facts. Jarvik's diseussion of gambling is 
largely discursive, and Edwards describes a series of experiments on gambling which 
endeavor to arrive at utilities. 

The success of the Symposium as such could be judged best from the discussion which 
is not published. As a publication, it is interesting for showing the diversity of activities 
under way, but unsatisfying just because of the diversity. As an issue of a periodical, this 
little volume would do very well. It lacks the cohesion to stand well by itself. 


The proofreading is rather poor; presumably the spelling "baracentric" on page 22 
is a typographical error. 


А. S. HOUSEHOLDER 


ни 


"же 


мьщ у Баай 
Lope 
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A TRIBUTE TO L. L. THURSTONE* 


Louis Leon Thurstone’s contributions to the development of psychol- 
ogy as а quantitative rational science are among the major scientific achieve- 
ments of the present century. Many of us here still remember the thrill of 
learning about his early developments in psychophysical scaling methods 
during the 1920’s. The psychophysical measurement methods developed in 
the latter part of the nineteenth century were used to measure the functional 
relation between the physical intensity of a stimulus, such as a light or a 
sound, and the psychological intensity of the sensation—the brightness of 
the light, the loudness of the sound. Thurstone was among the first to point 
out that these methods could be modified to construct a scale for measuring 
psychological qualities that had no measurable physical correlate. He de- 
veloped the law of comparative judgment and demonstrated that in con- 
junction with the method of paired comparisons it could be used to measure 
purely subjective attributes, such as the aesthetic merit of paintings or the 
strength of an attitude. 

In all his work, he stressed the fact that as long as we have merely a 
rule of procedure for analyzing data, we have no science. He insisted that 
every theory must be so precisely stated that one of the possible conclusions 
would be that the data collected were in disagreement with the theory. 

In particular for the law of comparative judgment, he developed the 
criterion of internal consistency for psychophysical scales. For linear scales 
this means that the distance from object ? to object 7 (as determined from 
one set of judgments “ greater than J”) plus the distance from 7 to k (as 
determined from another set of judgments “7 greater than k") should give а 
Sum in reasonable agreement with the distance from i to k (as independently 
given by judgments ' greater than К”). Such a statement must be true 
for all possible sets of three, so that if one has as many as ten or twenty ob- 
jects in the scaling experiment the number of checks becomes very large. 

This, of course, is merely one illustration of the application of the criterion 
of internal consistency. One of Thurstone’s important contributions was his 
insistence that each experimental and analytical procedure must contain 
such internal checks. н 

In addition to devising the theoretical framework for such psychological 
measurement, we remember Professor Thurstone for numerous applications 
of these methods to practical problems that were carried out by him and 
n behalf of the Psychometric Society at 


*A statement read by Harold Gulliksen 0! iu 


the annual meeting of the Society, September 4, 
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i ects 
his students. One of the early Mi gc iiem = аис 
of various movies оп attitudes o children; otl А s E, fis eee 
of various propaganda devices in changing attitu ез, ar init ur а 
i i ini eflected in newspapers over a period of years 
a pelo scii opened by Thurstone over thirty — 
is still developing both in its theoretical aspects and its uses. App em 

f these methods to measurement of intensity of attitudes and to the pr к 3 | 
D in of value systems have been or are being made. The ККЕ! 
scaling methods are important and powerful scientific tools. During the кошы. 
decades, prolifie and fruitful use of these methods in the SRERASDUNS a po 
fields as linguistics, к; cultural anthropology, political science a 

1 rill probably be seen. | 
еке» ме dana in psychology cannot be properly appreciated 
unless seen against the backdrop of psychological developments of the last 
half-century. Today it is taken for granted that aptitude and achievement 


tests can predict various types of academic and job performance with a use- 
ful degree of accuracy. Throughout this country, 


of persons take some millions of tests annually. How often do we stop to re- 
member that prior to 1900 there were literally 


no aptitude tests available 
for prediction of academic or job performance? Prediction of educability of 
children was one of the critical unsolved problems of democratic society. 
Commissions were appointed in various countries to study the problem. 
The Binet test was developed in France and was useful in predicting school 
achievement but was an unwieldy and cumbersome instrument. During the 
First World War, the Army Alpha and Army Beta tests were developed 
and used in the first large-scale mass testing program in the history of the 
world. L. L. Thurstone, as a young psychologist, was active in this testing 
program. 


Thurstone saw the inadequacies in the then widely accepted notions of 
"general" intelligence. He realized that new 


and more powerful methods 
must be developed to analyze the masses of data necessary for a thorough 
study of the different aspects of cognitive ability, or as we now say, the 
different factors, or primary mental abilities, He made a very simple change in 
the then current theory. In that each person was to be 
characterized by one num 


that it might take a great many numbers to describe t 


he person—one number 
for each of the primary mental abilities. 


He outlined his develo 
he knew and thus learned t 


some hundreds of thousands 


He tutored regularly, 
was the development 


human abilities. 
exts—the result 
n applied exten- 
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sively by Thurstone, his students, and. others, not only in analyzing the 
domain of mental ability but also in studying bloes in a legislature, schools 
of thought among teachers regarding curriculum content, classifying aller- 
gies, analyzing anthropometric measurements, organizing psychotic symp- 
toms, and so on. The factor methods are extremely powerful, and opportun- 
ities for their application in the study of human behavior seem to be limit- 
less. These developments and applications stem in large measure from Thur- 
stone's gift for seeing an important problem, defining it elearly, and then 
sparing no effort in his persistent search for a solution. 

Тыз development of testing over the past fifty years should be seen 
not only as a scientific achievement but also as a humanitarian accomplish- 
ment. Fifty years ago any young person who wished to enter on a given 
course of study had no alternative but to try, if he were permitted to, for 
months or even for years; eventually by trial and error he would succeed 
or be discouraged by repeated failures and cease trying. Now the inept 
student need not face the discouragement of tackling too high a goal, and 
the gifted student need no longer be dependent on attracting the attention 
of some influential person in order to obtain opportunity for advanced train- 
ing. The unknown person of talent can be identified and encouraged to pro- 
ceed with advanced training in some appropriate area. Such identification 
and utilization of various degrees of talent can be of enormous benefit to 
society and also a boon to the individual who derives added happiness from 
engaging in an occupation which matches his abilities. 

Fifty years ago no aptitude tests were available for such prediction of 
various aspects of human performance. That we can now see our way reason- 
ably clearly to this goal is in no small measure due to the theoretical and 
the practical contributions of Professor Thurstone to the field of psycho- 
logical measurement. His outstanding contributions to the fields of factor 
analysis and psychophysical sealing by no means exhaust the range of his 
achievements toward developing psychology as а quantitative rational 
science. He worked on developing personality measures, and on a quantita- 
tive rational learning theory. His Ph.D. thesis of 1916 constitutes one of the 
early attempts to develop equations of the learning curve. | 

On his retirement from the University of Chicago, his work continued 
to be as productive and ingenious as ever. At the University of North Caro- 
lina he was leading projects concerned with developing new psy chophy sical 
scaling methods and developing a set of novel ideas for obtaining objective 


measurements of personality characteristics. It is to be hoped that these 


ideas will not be lost, but that some research worker or group of research 
workers will develop and validate personality tests along the lines indicated 
by Thurstone. : . . д 
During more than а quarter century at the University of Chicago, in 
addition to making outstanding contributions in many scientific areas Pro- 
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fessor Thurstone trained large numbers of students who came not only from 
the United States but also from various foreign countries to study with 
him—to learn the techniques he was developing, the general principles of 
scientific investigation, and the principles of the quantitative rational ap- 
proach which he espoused. 

Louis Leon Thurstone has a unique position among psychologists of 
this century as an original research worker and as an inspiring teacher. 
Many of us who were privileged to know him closely found him a helpful 
and understanding friend. 

We, the members of the Psychometrie Society, feel it our particular 
privilege to pay homage to his memory. He conceived of this Society and 
its journal, Psychometrika. To his efforts more than to those of any other 
individual, both the Society and Psychometrika owe their present status and 
even their very existence. ә 

As psychologists, we feel that our past achievements and our future 
aspirations in the theory and practice of psychometries have been greatly 
areata e riae T a a nights, аай by ha andar 

reel d і 5 ideal: the development of psychology 
аза quantitative rational science. The greatest honor we can accord him as 
an outstanding scientist lies in our resolve to continue the development of 
psychology in the rigorous tradition he did so much to establish 
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THE SCALE GRID: SOME INTERRELATIONS OF DATA MODELS* 


Сіурв Н. Соомвзї 


UNIVERSITY OF MICHIGAN 


А Perhaps an appropriate subtitle for this paper would be “Some Spec- 
^q on the Interrelations of Psychological Methodologies." The Scale 
| rid is a name I have given to a model which presumes to define the under- 
ра continuities between such diverse areas as psychophysics, objective 
ae atitude studies including questionnaire and interview techniques, 
earning experiments, rating scale methods, essay examinations, and pro- 
ы instruments. The intent of the Seale Grid is to make explicit the 
чан similarities and differences of the methodologies among these 
various areas of psychological research. The increasing abundance of models 


and methods in all of these areas, with their associated nomenclature and 


pecialized vocabularies, makes a unification of them increasingly desirable. 
h, serious and intensive 


In some of these areas of psychological researc 
efforts have been made to construct models on a genotypic level to explain 
and predict manifest behavior. One thinks here, for example, of the area of 
signal detection in psychophysics and of objective test performance. In 
other areas the models are less explicit and tend to be on à literary level. 
However, even in these latter areas one may look at the methods of analy- 
zing data. Because there is always а model, at least implied, some of the ele- 
ments of the models are evident. is universe of models, 
explicit and implicit, abstract certain ts, and try to charac- ` 
terize them. 

Our starting point W. 
ogy. What we want is an abstrac 
of psychological observation. Let us begin by taking some 
what the basic abstract datum js in each case, 
eral definition. In some psychophysical experi 
viduals judge which one of several stimuli is t 
the experimenter subsequently does with the data, it is evident that he 
thinks of each stimulus as & point; the judgment of the individual is inter- 
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preted as an order relation on these points. Over many replications, the 
model may deal with a distribution of points for each stimulus and be a 
probability or actuarial model, but this is not the type of distinction which 
is at all important to us now. Whether the models are deterministic or actu- 
arial is not relevant to the fundamental distinctions between methodolo- 
gies I want to make. In fact, I want only to make the point now that in 
some types of experiments the manifest behavior is interpreted as an order 
relation between a pair of points, both of which are identified with stimuli. 

A different case arises, however, when an individual takes a mental 
test and passes some items and fails others. The use to which the behavior 
is put also suggests that it is being interpreted as a relation on a pair of 
points, but here one point is identified with an item and the other point 
with the individual. The point associated with the individual represents & 
measure of his ability; the point associated with the item represents its 
difficulty. The behavior of the individual in passing or failing the item is 
interpreted as an order relation on this pair of points, I 


, : am not here con- 
cerned with the numerical scores on 


| а test or even how a theory arrives at 
such a score from the basic datum. These are differences on a higher level, 


and I am here concerned only with differences on most the primitive quanti- 
tative level. 


When an individual is given an attitude scale and asked to indicate 
which items he will indorse, again the behavior is interpreted as a relation 
on a pair of points—one a stimulus, the ot 


n a] her an indivi ге the rela- 
tion 1s on a psychological distance between the = ени туз =й 
з pria the point Associated with the stimulus. If the point asso- 

е wi „the stimulus is “near,” in a sense defined by the model, the 
p associated with the individual, he indorses the item, otherwise not: 


cating whether the distance between 
tain amount. 


en Е " 

rating scale is an ordered set of id ipe iig oa holds if the 

is also rating scale behavior. If we ask an individual ie] e poe 

there are ina class it does not matter whether he M RUE many stu s : 

as the basic datum is concerned. The response, e.g., «35.» is ternal = 
, 
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a relation between one stimulus, the size of the class as perceived, and another 
stimulus, a real number. 

When an individual is asked whether he observed a light increment or 
not, the behavior is interpreted as a relation between a point identified with 
the individual, a threshold, and a point. identified with the stimulus, the 
magnitude of the increment. For a final example, consider an individual 
asked to judge which of two pairs of stimuli is more similar. Here the be- 
havior is usually interpreted as a relation between two distances; if each 
distance is interpreted as a point, then behavior implies a relation on a 
pair of points. 

All of these examples illustrate one important fact: behavior is made 
into data by interpreting it as an order relation between points or a relation 
on distances—both may more generally be regarded as a relation on а pair 
of points. An important distinction must be drawn between behavior and 
data. A datum is defined in this paper as а relation between points. That 
this is not a new idea is evident from a half-page note by Madison Bentley 
(1) in which he speaks of Stumpf, Wundt, Ebinghaus, Mach, G. E. Müller, 
and others who took the view that psychological measurement is a distance 
measurement, which is just a special case of а relation between pairs. 

We have been speaking here as if there were just а single distinction 
between behavior and data. Actually а threefold distinetion should properly 
be made. We may use the term behavior to refer to anything observable 
about the individual, raw data to refer to that which is selected for analysis, 
and data to refer to the interpretation of the raw data as an abstract relation 
between points. The first step in going from behavior to raw data, deciding 
what to observe, is a many-faceted problem which lies outside the scope of 
this paper. We are here concerned exclusively with the raw data and how it 
is interpreted as data in the sense defined above. I shall pursue this dis- 


tinction between the raw data and the data, illustrating it in detail shortly. 
In principle one could put any data in a matrix as follows: If the data 


were a relation on a pair of stimuli then a square matrix with rows and 
corresponding columns identified with stimuli (cf. Figure 1) could nicely 
accommodate the data. Each cell would contain an entry indicating the rela- 
tion between that corresponding pair of stimuli. Another experiment, where 
one member of the pair of points was identified with an individual and 
the other with a stimulus would require that the matrix of Figure 1 be 
expanded as in Figure 2. Thus an experiment in which one point of a pair 
of points was identified as a stimulus and the other as an individual would 
be entered in the left portion; another experiment In which the behavior 
was interpreted as & relation on a pair of points, both of which were identi- 


fied as stimuli, would be entered in the right portion. н 
Now if we go 8 little further and consider an experiment where the 


members of the pair of points are both identified with individuals, the 
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matrix of Figure 2 becomes as in Figure 3. According to this figure, а 
behavior observed in some experiments is interpreted as a relation i5 tens 
of points in which both points may be identified with stimuli, bot! E ae 
may be identified with individuals, or one point with a stimulus an i 
point with an individual. When the data are relations between stimu 
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points, the analysis yields the order of these points on 
bute or the location of these points in a multidimens 
viduals who made the Judgments or responses are not 1 
space. So we might call such a Space a Stimulus s 
we can talk about a space in which only individuals 
lation space, and a Space with both stimuli and peopl 


have the kinds of spaces in which psychological d 
ified in Figure 4. 


a psychological attri- 
ional space; the indi- 
ocated as points in the 
pace. Correspondingly, 
are located as a Popu- 
е as a Joint space. We 
ata are analyzed class- 
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tion in order to bring out the implications of the grid. I find it rather diffi- 
cult not to do both as they are so closely interdependent. So I shall first de- 
velop some of the theoretical ideas which will be most related to the inter- 
pretations which will follow. We shall consider some typical experiments 
which are mapped into Joint spaces and some which are mapped into Stimulus 
Spaces; then we shall see what the difference is between them. The char- 
acterization of this difference will constitute one dimension of the Scale Grid. 


Model Underlying The Scale Grid. 


Behavior on a mental test is interpreted as a relation on а pair of points, 
one of which is associated with the individual and the other associated with 
the item or stimulus. Such behavior is mapped into data which, when ana- 
lyzed, yields a Joint space with both stimuli and individuals located in it. 
What is the primitive operation here? The test item was conceived of as 
having a certain difficulty and the individual was, in effect, asked to compare 
his ability level with the difficulty level of the item. In a psychophysical 
study of the thresholds of individuals, the same is true, e.g., the individual 
is asked whether he perceives the stimulus, yes or no, and the behavior 
is interpreted as a relation on a pair of points, one of which is associated 
with the threshold of the individual, the other with a stimulus magnitude. 

On the other hand, in a psychological study designed to measure heav- 
iness of weights, length of lines, brightness of lights, or what have you, 
what is the primitive operation? 'The stimuli are conceived of as points on 
an attribute continuum, and the individual is asked which of the stimuli 
is greater. The behavior is interpreted as data on pairs of points both of 


which are associated with stimuli. Analysis of the data locates the stimuli 


on a scale, but no attempt is made to locate the individual as a point on 
the scale. The result is a Stimulus space. 
Suppose I have some attitude statements about the church. I want 
to scale the items and then measure people's attitudes with them. The first 
Step is to scale the items, во We ask individuals to evaluate the items as to 
which is more pro-church. A data matrix is constructed which is эу 
to yield a Stimulus space, say & one-dimensional scale, with the items locate: 
as points on a continuum. We note that the individuals were asked to nen 
uate the items with respect to where the items were on the ARR 
not with respect to any point on this continuum which ор ec ue 
the individual. Having scaled the items, we turn around and as e 3 i- 
vidual which items he indorses. When we analyze these data, ye n up 
with the individuals located on the same ашын a ate a. a 
the experimenter gave the individuals а шее е к ш or sce 


i zed to evaluate the items wi 
Nei caedit ding to his own attitude toward the church, so 


the continuum correspon 
now we are in а Joint space. 
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I could go on with examples from conditioning experiments or woo 
in perception, etc., but will not take the time for it. The important t ie 
is to see the essential difference between behavior which is mapped into 1 
Joint space and behavior which is mapped into a Stimulus space. In m 
the experiments, there are always both individuals and stimuli—what is 1 
that determines whether an experimenter maps his experiment into a Sti- 
mulus space or into a Joint space? | А 

If you go back and look at experiments with this question in mind, it 
becomes obvious that the experimenter puts his experment in a Joint space 
or a Stimulus space according to whether he regards the individual as having 
evaluated the stimuli with respect to a point corresponding to himself, the 
individual, or whether the individual evaluated the stimuli with respect to 
an attribute. I have called these two kinds of tasks, task А and task В , Te- 
spectively. Task А may be described as evaluative, having to do with the 
relation of stimuli to the individual himself. Task В may be described as 
substantive, having to do with the nature of the stimuli per se. 

I formalized this distinetion between task A and task B in the following 
manner. In all experiments, both the individuals and the stimuli are points 
in a space, but in task А the points associated with the individuals are inde- 
pendent of the points associated with the stimuli. Whereas in task B, where 
the individual is evaluating stimuli with respect to an attribute, the point 
associated with an individual is completely dependent 
sociated with the stimuli he is evaluating. 
this now, but essentially what we have i 
with just its two extremes n 
vidual's point from those of 


imuli, corresponding to task А and task B, respec- 


I have taken а an intuitive notion of one 
dimension of 


just a few words about a 


ion space; we would find an equi- 
ust 'S of stimuli and individuals in the 
argument and analysis just made. The re i 


stimuli judged, whereas the reverse holds for 
We have here two of the dimensions of th 
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two others which сап be used to characterize the data within these major areas 
of Joint spaces, Stimulus spaces, and Population spaces; but I will say nothing 
further about them here as they are not relevant to the interpretations I wish 
to bring out. I will only say that something of their nature is described in my 
early monograph on theory of psychological scaling (2). 
; If we take the two dimensions which we already have, they suggest a 
ourth type of space, called a. Field space, in which the points associated with 
stimuli and those associated with individuals are completely mutually depend- 
ent. 'This gives us a two-dimensional Scale Grid as illustrated in Figure 5. 


Population 


Dual Task B 


Stimulus 


Dual Task A 


Task A Task B 
FIGURE 5 


The Scale Grid 


Some questions naturally arise as to just what might be the significance 
of all of this and just what this Field space type of behavior is. While we know 
Population, or Joint space, this Field space was а 
and it is not immediately obvious 
tion, I shall suggest а 


What goes into a Stimulus, 
consequence of our analysis of the others, 
What significance it has. In order to answer this ques 
psychological interpretation of the Scale Grid. 


Scale Grid 


I shall first point out what I consider to be the psychological processes 
involved in collecting data in a Stimulus space. Then by virtue of the duality 
between stimuli and individuals, a dual interpretation for Population spaces 
will be made and certain implications of duality pointed out. From these two 
kinds of spaces we shall move in one direction and get Joint spaces and in the 


other direction to get Field spaces. 
Consider what is involved whe 

individual, the subject, is asked to ma 

to an attribute (task B). He is given a se 


Psychological I nterpretation of the 


n we collect data in a Stimulus space. An 
ke judgments about stimuli with respect 
t of weights and asked about their 
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felt-heaviness, a set of tones and asked about their pitch or а и 
objects of judgment in this situation have many measures; eac 05]69 Fami 
measure on each of its many attributes—e.g., color, size, heaviness, , 
т ic quality, etc. 

ү ж ы Ашыт ennd to points having several iE epi 
and individuals instructed to select one of these components and evaluate the 
stimuli with respect to that attribute. Let me say it again and contrast the 
difference—the stimuli are points on many attributes, the individual Gomes 
with an attribute in mind but no scale position of his own from which to 
evaluate the stimuli. In an exaggerated sense, for a Stimulus space the stimuli 
have provided the points, and the individual has provided the attribute. These 
then are the respective functions of stimuli and individuals in generating data 
in a Stimulus space. The behavior observed is ultimately converted to measures 
of the stimuli on the attribute. The behavior observed may run the gamut of 
paired comparisons, rating scales, or free-answer protocols—these differences 
are not relevant; here. The important thing is that the behavior observed is 
interpreted as data which are relations on the stimuli with respect to the 
attribute. The analysis then leads to measures of the stimuli only. 

Let us now exercise the duality relation and consider what the process 
must be for a Population space. When we reverse the roles of stimuli and 
people we must have a group of individuals, each possessing measures on many 
attributes just as the stimuli had in a Stimulus Space. An individual thus 
corresponds to a point with many components. Then we must have stimuli 
which, carrying through the analogy, must be instructed to select one of these 
components and evaluate the individuals with respect to that attribute. In 
the Population space the individuals provide the points, and the stimulus 
provides the attribute. These then are the respective functions of stimuli and 
individuals in a Population space. What kinds of behavior do psychologists 


observe in which these are the respective functions of the stimuli and the 
subjects? I would say that certain 


у : ‚ Speaking category-wise, the most 
typical are rating scales and free-answer protocols. 

The rating scale method is the questionnaire item with a number of 
ordered alternatives—an example 


from a questionnaire used on soldiers during 
the war is the following: 
Do you ever get so blue and discouraged that you wonder whether 
anything is worth while? 


a) Hardly ever 
b) Not so often 
с) Pretty often 
d) Very often 
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Such a procedure is formally equivalent to asking an individual to judge 
weights as being light, medium, or heavy. But one experiment is in a Popula- 
tion space, the other in a Stimulus space. 

Free-answer protocols are illustrated by the essay examination, the open- 
ended questionnaire, and the interview. The individual is asked a question 
which in principle specifies an attribute, e.g., How do you feel about the farm 
policy? The individuals who are asked this question are playing the role of 
stimuli being evaluated with respect to an attribute. The protocols which 
emerge are analyzed for relations between the individuals, which constitute 
measures of them on this attribute. This is formally equivalent to the use of 
individual’s evaluations of stimuli with respect to an attribute, which leads 
to information about where the stimuli are on the attribute selected by the 
individual. 

One immediate consequence of this duality between Stimulus spaces 
and Population spaces is that any method for collecting or analyzing data 
constructed for either one of these spaces immediately becomes а potential 
method for collecting or analyzing data for the other. Thus we have, for ex- 
ample, Thurstone’s Law of Comparative Judgment constructed for analyzing 
the judgment of individuals about stimuli and arriving at a Stimulus scale. 
Immediately there is implied the dual method of having stimuli make paired 
comparison judgments about individuals—as yet I have seen no good way of 
getting stimuli to do this. However, there are variations of this basic method of 
Thurstone’s: the Method of Successive Intervals and the Method of Equal 
Appearing Intervals are used for constructing Stimulus spaces which do 
transfer completely to Population spaces. To transfer the Method of Succes- 
sive Intervals to Population spaces you need to have stimuli sort people into 
piles. I suggest this is exactly what is done by those questionnaire items with 
multiple alternatives, e.g.; from strongly agree to strongly disagree. Abstractly, 
we can look upon such behavior as stimuli sorting individuals into piles. In 
the Method of Successive Intervals the instructions to the subject are dual- 
istically equivalent to the writing and editing of an item for a questionnaire 
Or essay examination. I find it strikingly curious that we frequently tend to use 
five degrees of indorsement or five ordered steps in the alternatives, whereas in 
the Method of Successive Intervals we have individuals sort items into as 
many as eleven piles. Whether there is а profound reason behind this, or 
whether it is unjustified adherence to tradition, I am not sure. 

Just as Thurstone's methods for Stimulus scales are transferable to 
Population spaces, so also are methods designed for the analysis of 7 E 
Population spaces alternative methods for Stimulus spaces, €-8-, Lazarsieid 5 


methods of latent structure analysis could be used for scaling stimuli ts 
Stimulus spaces by reversing the subscripts which identify stimuli Wm ^ 
viduals, and obtaining the appropriate kind of judgments from indivi ы s. 

We see here, in fact, the relation of certain methodologies of the psycho- 
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physicist studying Stimulus spaces to those of the social psychologist pe бе 
Population spaces. What the first makes people do to the stimuli, t ae. d 
makes stimuli do to people. Their methods of doing research, collecting, uu 
analyzing data are formally isomorphic with the roles of stimuli and people 
reversed. Surely, with reference to methodology, whatever one develops BUB 
gests а dual development for the other. Each delineates an attribute on which 
the objects of judgment are to be evaluated. In psychophysics the experimenter 
does this through his instructions to the subject; in questionnaires and essay 
examinations the experimenter does it through his careful writing of items. 
АП the various experimental controls developed in one context, again, in 
principle, transfer to the other context with the reversal of roles between 
stimuli and people. E 
The greater status in measurement of psychophysics is due, at least in 
part, to the fact that an individual ean compare two stimuli directly, whereas 
a stimulus cannot compare two individuals directly. We are much happier 
with the judgment of an individual as to which of two attitude statements he 
prefers to indorse than we are with the judgment of which of two individuals 
indorses a given statement more strongly. The reason for this is very simple. 
When we ask an individual which of two stimuli he prefers, we assume he has 
an implicit standard of measurement that is an ordered scale applicable to 
both stimuli. If we wish to compare two individuals as to which indorses & 
stimulus more strongly, we have to assume not only that they each have an 
implicit interval scale but also that the scales have the same origin and the 
same unit of measurement, Thus, if individual A says he would pay $10 for a 
picture and individual B $5, how do we know but that A has less value for 
money than B? Once we have made the assumption of an interpersonally 
comparable interval scale, there is no sense in reducing the data to a paired 
comparison—that would be throwing away i 
This argument can be sum 
judgment of one person are 
the standards of two his-might well be the con- 
sideration that underli 1 1 


With Stimulus and Population s 
we turn briefly to Joint s 
together, jointly specifyin 
own measures on these 
working an arithmetic pr 


paces mutually described and related 
paces. Here both stimuli and individuals come 
£ what the attributes will be; both have their 
attributes. For example, consider an individual 
oblem. The arithmetic problem is represented by 
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a point with measures on one or more components. The individual is similarly 
represented by a point with measures on these components. The response 
of the individual to the stimulus will be information about the relation of 
these two points in the Joint space. 

By defining what this information is in different ways, one gets Guttman 
scalogram theory, test theory, one of Lazarsfeld's models, or my unfolding 
technique for the analysis of preferences. All of these are just different models 
for what a response on the phenotypic level means in terms of distances 
between pairs of points in the Joint genotypic space. [The relation of these 
various spaces to the classification of methodologies contained in (3) should 
be pointed out. The methods of collecting data which apply to Joint spaces 
are classified in Quadrants I and II, and the methods which apply in Stimulus 
Spaces or in Population spaces are classified in Quadrants III and IV.] 

In Joint spaces all the psychophysical methods for analyzing experiments 
concerned with thresholds, as distinct from those concerned with measuring 
only the stimulus magnitudes, are present. The individual in such experiments 
18 regarded as having а threshold on an attribute, such as his sensitivity to 
light or his ability to discriminate pitch. This characteristic of the individual 
corresponds to a point in the genotypic space, which we have called his ideal 
on that attribute. The stimulus then is an increment of light or a difference 
between two tones, and the individual is asked whether he observes it. The 
stimulus is then also represented by a point in the space. The response of 
the individual is a formal relation on the pair of points in exactly the same 
manner as passing or failing an arithmetic item. 

The data obtained from neurotic inventories and interest inventories 


are typically mapped into Joint spaces. For example, an individual is asked 
h he is to answer yes or no. The individual 


a question like “Are you shy?" уме 

is presumed to prossess and recognize his particular amount of shyness— 
this corresponds to a point in the genotypic space, which is his ideal. The 
question “Are you shy?” with the alternatives yes or no also corresponds to 
a point in the genotypic space which is that amount of shyness the individual 
feels he should have to say yes. This amount of shyness is formally equivalent 
to the difficulty of an arithmetic problem, the increment of light, or the 
difference between two tones in the preceding examples. Again the individual’s 
response to the question is interpreted as а relation between the respective 


points. 

That the data obtained from individual’s indorsements or preferences 
between attitude statements may also be mapped into Joint spaces is too 
obvious to need further description. Most learning experiments are mapped 
into Joint spaces. A conditioning experiment, for example, is like an objective 
test given backwards: а combination of unconditioned stimulus and con- 

em, and eliciting a conditioned 


ditioning stimulus may be thought of as an it z 1 е 
response is equivalent to passing an item. Then the most difficult item m 
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the test is presented first, i.e., the first presentation of PH енш 
'and unconditioned stimulus, and the individual usually fails it. 4 "pine. 
takes place each successive presentation is essentially an bcd E sies 
items are so easy that the individual passes them all. It is тае і ея 
that the conditioning test has a different method of scoring from the [n s aie 
mental test, e.g., the number of items taken to reach a certain num " a 
items passed successively. One wonders why most objective tests shou А 
have а different convention. I do not object to different conventions, I jus 
ik know what the logic behind them is. "o 
d E we have all these superficially different kinds of behavior: prim 
tests, certain psychophysical experiments, neurotic inventories, euge 7 
questionnaries, attitude scale studies, and conditioning experiments. 
tend to develop their own methodologies and their own vocabulary—but 
all are formally isomorphic and hence their methodologies transferable from 
one to the other. A model for analysis of one of these kinds of data with a 
particular distance function, for example, immediately raises the question 
whether it does not also constitute a theory about each of the other seemingly 


different kinds of behavior. There are, of course, differences in the character- 
istics of the data 


one gets in these areas. In some areas experimentally in- 
dependent replica: 


tion is possible, in others not; in some areas the stability 
of a point associated with an individual or 


Before going on to a 
should digress f 


pped. 
The act of a psychologist in putting his experimental data into one of 
these spaces (Joint, Stimulus, or P 1 
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into data that we sometimes neglect any mapping but the conventional one. 
While data is obtained from behavior by interpreting the behavior as a relation 
between points, it is up to the interpreter to decide what to identify as 
points and to define the properties of the relation. 

Consider a study on nationality preferences. Let each subject make 
paired comparison judgments as to which nationality he prefers. In 
Thurstone’s well-known study (5) such data were analyzed by the Law of 
Comparative Judgment and a scale obtained with the stimuli ranging on a 
one-dimensional continuum from most preferred to least preferred by the 
group as a whole. When the experimenter does this, he is regarding “ргеѓ- 
erability" as an attribute of stimuli and is saying that the individuals made 
task B judgments, substantive judgments, about the stimuli. The behavior 
is interpreted as an order relation on pairs of points, both of which are stimuli. 
He is saying the behavior belongs in a Stimulus space, and proceeds to con- 
struct а stimulus scale. . 

On the other hand, one could take the identical experiment and put it 
in a Joint space. In doing this, one would be assuming that the individuals 
were also points in the space and that their behavior is to be interpreted as 
an order relation on distances of the stimuli's points from the individual's 
point. Thus, the behavior is being put into a Joint space instead of a Stimulus 
space, and analysis of the data by multidimensional unfolding would yield 
a solution with both stimuli and individuals in the space. 

There is nothing intrinsically correct about one of these procedures 
or wrong about the other. In the first instance, analyzing the data in a 
Stimulus space, one’s problem is essentially that of amalgamating the prefer- 
ences of individuals to arrive at a single preference scale, which in some 
sense best represents all the individuals. This is the problem of social utility 
or social choice. In the second instance, analyzing the data in a Joint space, 
one’s problem is that of discovering the latent attributes underlying national- 
ity preferences from which an individual’s preferences could be derived. It 
might be parenthetically remarked that these two solutions would bear a 
certain interesting relation to each other, this relation has been developed 
in two previous publications (2, 4). . Es 

Here we have taken an example of behavior and made the transition 
into two different kinds of data, analysis of which yields different results. 
We usually overlook this step that we take between behavior and data 
because this step, at least in some areas of research, is so conventional and 
immediate. Everyone can usually agree on what is or 15 not the right answer 
to an arithmetic problem; when an individual Says this weight is heavier 
than that one, everyone usually agrees he means this weight is higher on ап 
attribute of felt-heaviness than that one. But when we have an individual's 
answer to an essay examination question or his answer to an open-ended 
questionnaire item, we speak of “coding” them. This is the process of con- 
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verting the behavior to data by processing it through the mind of another 
person to get statements of magnitude or relations—these data are what are 
analyzed. It is important to note that what one analyzes is always data, 
not behavior. 

This distinction between behavior and data now becomes an even more 
important and relevant distinction as we turn to a psychological interpretation 
of Field spaces. To arrive at this interpretation we move along two dimensions 
of the Scale Grid simultaneously. In going from Joint to Stimulus spaces, 
the point associated with the individual became dependent on the stimuli 


If you wanted to observe such behav} 
wante vior what would yo ? id 
present an individual with a stimulus that wa: PIU diues 


it would not arouse а 5 of such an ambiguous nature 
n s. i Eam 
; ФО; асе in individuals. me 
time the individual w R uals. At the ва 


points, or the distance 
done by definition, is 
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In a Stimulus space, the individual was instructed to ask of the stimulus 

how heavy it was, or how esthetically pleasing it was, etc. An attribute 
was explicit, and so the behavior could be interpreted as magnitudes on an 
attribute and thus made into data. Analysis of such data leads to conclusions 
about interstimulus differences. In a Population space the roles are reversed: 
a stimulus comes to the individual and asks him how he feels about the farm 
policy. Again an attribute is explicit. The behavior is interpreted as magni- 
tudes on an attribute, analysis of which leads to conclusions about inter- 
individual differences. 
_ , Ina Joint space both interpretations are possible because the behavior 
is interpreted as data on a relation between individuals and stimuli. In a 
Field space the point associated with the individual has merged with the point 
associated with the stimulus. The behavior is information about this point 
ша psychological space, a point in which the subject and the stimulus are 
inextricably identified. 

The care given in Population spaces to selecting items for a questionnaire 
Or essay examination in order to ask every individual the same question, and 
the care given in Stimulus spaces to phrasing instructions to the subject in 
order that he evaluate all the stimuli on the same attribute is now exercised 
in Field spaces so that precisely these effects will not occur. Every effort is 
made to insure a setting in which the stimulus will not suggest a particular 
attribute space, and every effort is made in the instructions to the subject 
not to suggest а particular attribute space. Herein lies both the strength and 
Weakness of Field spaces. The protocol that emerges now constitutes a 
Stimulus to be evaluated—so it is а stimulus to be located in a Stimulus 
Space. In order to convert this protocol, this behavior, into data certain 
problems need to be solved. One is: what is the attribute or attributes which 
underlie the behavior? This is a new problem which had not previously 
arisen for any of the other spaces. Here now we have a protocol which is to 
be converted into a measure on some attributes. The first problem is: which 
attributes? This problem arises because there was not deliberately built into 
the stimulus nor into the individual constraints or instructions which would 
Provide a simple answer. М 

It is immediately obvious that behavior in this area does not lead to 


interindividual comparisons, because there has been no instruction to the 
Stimulus, no built-in device by virtue of which it can be assumed that a 


stimulus has evaluated each individual on the same attribute. If one person 
exhibits guilt feelings and another does not, one cannot conclude 8 d 
person has less guilt feelings unless one can assume the stimulus was suc 

as to make every individual reveal his guilt feelings. In this case, of course, 
such data could be put back in Population spaces, and interindividual com- 


parisons would be possible. M | i 
It may help one to recognize and understand this problem if we point 
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hat it is like having the answer of an individual to an essay examination 
а I Љеп you do not know what the question was. Consider, then, 
ee cca answers of several individuals, each to an unknown question; 
the problem is to decide which individual's answer represents more of са 

ing than another's. It seems to me the problem is meaningless if the questi я 
Answered by an individual has been left up to him to select, hence «p 
individual has perhaps selected a different question. One could say, wel 
can evaluate their relative command of English, or their vocabulary ine 
or their handwriting if it is a written protocol. This is entirely correct, Е 
course, and amounts to saying this is the common attribute which t ry 
stimulus aroused in all individuals, hence they may legitimately be compare : 

- This puts the behavior into a Population space, not a Field space. What 
am talking about are those aspects of the behavior which are not attribute 
controlled and hence belong in a Field space. 

Understandably enough, there have been instruments constructed, 
called projective instruments, which seek to avoid this particular problem. 
For instance, there are test instruments in which a picture suggests an 
attribute; the individual is asked to write a story which is presumed to 
reveal where he is on that attribute. Examples are Proshansky’s Labor TAT, 
Johnson's Anglo-Spanish TAT, and toy play with negro and white dolls. 
If these instruments succeed in their purpose, then we have the stimulus 

coming to the individual with an attribute in mind and asking the individual 
where he is on it. These instruments then, if successful, are formally the 
same thing as a rather subtle essay examination or an interview by a laborer, 
a Mexican, or a Negro. The data that are 

space rather than a Field space, and interindividual comparisons are logically 
permissible. Such instru 

in the sense of belonging in Field spaces. A further question then arises 
as to whether or not th complish their purpose. If they 
perimenter subtly built into the 


rawing false conclusions. 
Let us assume that the first problem 1 


answering different questions, the 
behavior cannot be taken to reflect interindividual comparisons on a common 
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attribute. On the contrary, and therein lies both its importance to psychology 
and its weakness as data, the behavior reflects intra-individual comparisons. 

A protocol in a Field space reflects a point in a psychological space. 
When we know enough to interpret the protocol as a measure of that point 
ina known attribute space, then we shall be able to make comparisons between 
the points. I suspect that these will be comparisons on some hyperabstract 
attributes which will reflect intra-individual dynamics. The problems which 
must be solved to reach this stage are what I would regard as our ultimate 
measurement problems. Field spaces are а maximally significant domain of 
behavior. It is the area that reflects intra-individual differences to a degree 
that no other area does. There are fascinating and important problems for 
psychologists here. It is my thought and hope that the Scale Grid will help 
to delineate more clearly the basic measurement problems involved. 
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EFFICIENT ESTIMATION AND LOCAL IDENTIFICATION IN 
LATENT CLASS ANALYSIS* 


Ricuanp B. МсНосн 
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, Estimators which are efficient in the sense of having minimum asymp- 
totie variance are obtained for the structural parameters of Lazarsfeld's 
latent class model of latent structure analysis. Sufficient conditions for 
the local identification of the structural parameters are also presented. 


1. Introduction 


A psychologieal test of p items or subtests will be denoted by 
(0, , 25 , +++ , 2). Data obtained from the administration of a psychological 
test can be processed in different ways. For example, in Lazarsfeld's latent 
Structure analysis (14) the test data are treated qualitatively, i.e., as measure- 
ments on a nominal or at most an ordinal scale. On the other hand, in classical 
Spearman-Thurstone factor analysis test data are treated quantitatively, 
i.e., as measurements on an interval scale. Despite this difference there is a 
feature common to these two approaches, viz., the view that the ath exami- 


nee's observed test behavior (Tia , tee , *** » Ura) May be thought of as 
being determined by his status (fia , ёа, "^^ > Exa) on A traits or factors 
(& , £& , +++ , &) underlying the test. That is, observed test behavior (referred 


to as phenotypic or manifest) is viewed as only an indicant of, not as a direct 
Measurement of, the underlying traits (referred to as genotypic or latent). 
[Among others, Coombs (2) and Stevens (13) offer justification for this 
&pproach to psychological measurement.] 

Fundamental to this kind of test analysis are the concepts of structure, 
model, and identification. In section 3 below, the general formulation of 
these concepts due to Hurwicz (5), to Koopmans and Reiersgl (6), and to 
Reiersgl (11) is applied to a partieular model of Lazarsfeld's latent structure 


analysis, viz., the latent class model. 
In addition to (a) the pre-statistic 


model, other problems which arise in t 1 b é Ј 
psychological test include (b) the efficient estimation of the identifiable 


parameters of the model, and (c) the scaling problem, i.e., the assignment of 
trait measurements (Eia , £20, `7 » ёла) tO each examinee. Section 3 is con- 


al problem of identification of the 
his approach to the analysis of a 
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cerned with problem (a) and section 4 with problem (b). [Problem (с) Ж 

been discussed for the latent class model by Lazarsfeld (14)]. Section 5 pr E 

sents a statistical test of the goodness of fit of the structure, and in section 

a numerical example is presented to illustrate the mechanies of computation. 
Section 2 provides a resumé which makes certain features of Lazars- 

feld's presentation (14) explicit in a form and notation convenient to the 

remainder of the discussion. 


2. The Latent Class Model 


Three distribution functions and an assumption concerning one of them 
constitute the essential ingredients of the latent class model. 

(i) Let G(x, , z» , --- , 2,) denote the distribution function of the p 
observable (manifest) random variables corresponding to the p items or 
subtests of the test. In practice, the subtests or items are usually dichotomous, 
i.e., the response of the ath subject to t; , denoted by t;a, equals 1 for à 
positive response and 0 for a negative response (i 


= 1,2, ... , p). Since the 
domain of the ith random variable, т; , is 0 and 1, then G(z, , 2, , +++ , £p) is 
à p-dimensional binomial distribution function, Let the set of parameters 
defining the test response distribution, G(x, , Yo, +++ , p), be denoted by 


{Gisis-+-i,}. These parameters are called manifest marginal probabilities. There 


тро 812, 7*7, grag . By definition, 
^ aud = 1], grs... = РИ = 0,2 = 1, = 
*, $5 = 0}. 

; the test may be interpreted as a random 
Ey ttt, € of p components—no effort is made to 
atterns. This may be contrasted with the 


are 2” of the elements iis «is, VIZ., gio 

fi. = Plan = 1, 2, = l, + 

2, = 1}, and so forth to 9.5 = Pix, = 0, 2, = 0, -- 
In the latent model, then 

vector (z, , a, 


i 


“++ , &) denote the distribution f. i f the À 
unobservable (latent) rando; i ARLON D 


(14) is generally to construct a pure test, i.e., a test 
it will be assumed here that ^ = 1. That is 


the trait £ is postulated to be a one- 
Stribution (Е). 
stricted to a point set of two elements, 
n general, the domain of Е consists of а 


Ё ents are termed latent classes. The generic 
class will be denoted by с (с = 1,2, ..., ). For example, in the numerical 
illustration in section 6, where the latent trait Ё is creativity in machine 
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design, the case of y = 2 is considered, viz., latent class c = 1 of creative 
machine designers and latent class с = 2 of non-creative machine designers. 
(By means of a chi square test of goodness of fit, however, as noted in sec- 
tion 6, a possible inference for this example is that y > 2). 

А Оп the basis of this qualitative interpretation of the latent trait, (8) 
is a one-dimensional multinomial distribution function, specifically a one- 
dimensional y-nomial. Let the set of parameters defining the latent trait 
distribution F(£) be denoted by {fe}. These parameters are termed latent 
marginal probabilities. There are y of the elements fe , viz., fi, fs, 7 fs 
By definition, f. = Р{# = c]; (c = 1,2, ---, у). 

For this model, the conceptual measurement £, of £ for the ath examinee 
(i.e., £, is conceived of as the value that would be obtained for the ath sub- 
ject if the latent trait £ could be measured directly) signifies an assignment 
of the ath subject to one of the latent classes c = 1, 2, +-+- , y. This problem 
of scaling, i.e., of drawing a latent inference from the observed test response 
to the unobserved trait, is problem (c) eited in section 1. 

: (iii) The final distribution function of the latent class model is the distri- 
bution of the subtests x, , £a , +++, x, for fixed £. Let G(x, , 2, +++, 2, | £ = 0) 
denote this conditional distribution function. 

This conditional distribution is important because it makes explicit 
the relation between the observable variables x; taken jointly and the un- 
observable ё. For example, one of the 2” parameters defining this p-dimen- 


sional binomial distribution G(x, , te, --- ,z, | E = c) is the probability of 
& conditional joint positive response, say 012... . Defined for the ath subject, 
this is Dione = P{tyq = 1, Xa. = 1, ++- , 4, = 1| E = c], indicating 


that the ath subject’s test behavior depends probabilistically upon his status 
on the trait ё. 
Instead of dealing with the joint conditional distribution G(x, , % , 
t, x, | Е = c), however, it may suffice to deal instead with the marginal 
conditional distributions G(x, | E = с), б( | E = 0), +++ , GG = с). This 
great simplification is possible if the psychological test has been constructed 
in such a way that 
Merta ESO (2.1) 
= Ga, lE = 9G |E=9 ++: GG, | = 0), 
e jointly statistically independent for 


‘or a specified population of examinees, 
pure (14). That is, if (2.1) holds, 


le. in case the subtests or items an 
fixed trait level or latent class £ = c. F 


a test with this desirable property is called t d 10l 
each ж, involves a single common trait £. Without this factorability condition, 


i i i veral common traits 

the subtests or items would involve ambiguously, several 
$r, é sé & (1 <A € p). In all that follows, Lazarsfeld's postulate (2.1) 
я : == 1 H p 71 T i 
ofa pure test will be assumed. The left and right sides of (2.1) will therefore 


be used interchangeably. 
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Because of their importance in the latent class model, the conditional 
distribution functions G(x; | ё = c) of the various items, or subtests, are 
given the special title of item trace functions (14). These G(x; | H = c) aro 
one-dimensional binomial distributions. By definition, the generic defining 
parameter iS g;,, = P{x; = 1| = c]. The g,,. are referred to as item con- 
ditional probabilities. 


To summarize, the latent class model involves the three distributions: 


(0 G(z , T2 , -+ , т„), defined by the set of 2? parameters TERES 
(i) F(£), defined by the set of y parameters {/.}; 
(iii) Ga, , £2, --- , £p | E = c), defined by the set of рү parameters 
{йез @=1,--+ ,p;e=1,--- , 4). 


TABLE 1 
Schematic Representation of the Latent Class Model 
One latent variable 


Domain of 6 (i.e.,, y latent classes) 1 oes 
Latent marginal probabilities " 


p manifest variables | Domain of x, 


(items or subtests) x. |(i.e., dichotomy)| 


Table 1 gives a convenient scheme for re i 
Presenting F(t) and G(x, 3, , t> 
1, | £ = c), as defined by (f.] and ы respectively ng 
The three distributions (0), (ii), and 
equations for the latent class 


(iii) are related by a set of 2” 


model: 
fize = > 1.0.92; pie 05:6 5 
9i» = Ly теда i Gore , up 


fiz-.5 = x £91.93. ttt Oie 


, 
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where the summation is from c = 1 to y. The set (2.2), the accounting equa- 


tions, follows from the fact that the distribution G(z; , 22, -:- , T») can be 
uniquely determined from the distributions F(Z) and G(T: 25, T| E= 0) 
by summing the product F(£)G(zi , T2, °° * , Tə | Е = c) over the domain of £. 


А It should be noted that of the 2? manifest marginal probabilities only 
2" — 1 сап be functionally independent since 


У Qiii = i (2.3) 


where the summation is over all the 2? values of ù à; +++ t а domain which 
will be denoted by D. Similarly, of the latent marginal probabilities only 
y — 1 can be independent because 


X fus Т. 


с=1 
Finally, the item conditional probabilities are related in general as 
gie Tu. T. 
3. Local Identification of the Structural Parameters f. and gii 

The parameters f. and gi, (6 = 1, °°: $36 = 3, ee DIO Table 1 
are called structural parameters for the following reason. A structure, denoted 
here by S? = (FE), G(x, ,22 77,24] = 9}, consists of the combin a 
tion of particular, concretely specified distributions F*( and G*(n , 2, 

2 1.28 | £ = c). Equivalently, in terms of the defining parameters, S = 
{fe 09) @ = 1, +++ ире = boi) That is, a structure is the combina- 
tion of a set of particular numbers f? and gi, for the parameters f. and 
9i;- . Table 4 presents an example of an estimated structure. 

From (2.2) it is clear that a particular structure {fe , ате} uniquely 
determines the set of parameters (саев) defining the distrubution function 
Gay ‚у кей ДШ), of the manifest subtests. The set {gatasi is said to 
be generated by the structure 5°. Because each different (permissible) set 
of specific numbers fe and gi. constitutes a different structure S^, it is 
natural to consider the class C — {S} of all such structures—this is the 
technical definition of a model. By definition, the latent class model C is the 
class of all structures {8} = if.» д.) conformable with the specifications 
of section 2. By contrast, an individual structure S? is а particular realization 
of the model, C. 

It will be assumed in what follo 
Then a problem of identification arises 


ws that there is at least one structure. 
ina natural fashion. That is, a particular 


structure {f?, g*..] generates опе and only one set of manifest parameters 
(93.;.--.;,]. Hence the very practical question, called the identification question, 
is then posed, viz.: Does the converse proposition hold, i.e., can EE 

be generated by only the structure. lUe gi)? Ш 50, then a given set of 
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manifest marginal probabilities {g?,;,...;,} determines in principle one vm 
only one structure [f? , g2..]. If this were to occur, the latent шч itn 
C — [S] would be said to identify (uniquely) the particular PRERE 
or the structure S° would be termed (uniquely) identifiable by the la 
5 1. 

кы ee condition for identifiability of the structure 5° or set of 
structural parameters f? and g?., is clearly that the number of independent 
pieces of manifest information be at least as great as the number of ide- 
pendent latent unknowns, i.e., the number, 2” — 1, of independent manifest 


parameters 5,;,...;, be at least as great as the number y — 1 + py of inde- 
pendent structural parameters f? and fie, ог 


2° > p+ 1). (3.1) 

The answer to the general identifieation question may be that several 
Structures besides S^ generate the Same manifest test distribution, i.e., 
the strueture 5° is not (uniquely) identifiable by the latent class model. 
However, a weaker form of identification, called local identification, may 
exist, viz., it may be that other structures generate G"(x, , z, , ... , ty); 
but none of these lies in the neighborhood of 5°. (For convenience, the super- 
script “0” will be omitted in the following.) 
THEOREM 1. If specifications (i) through 
t class model, then the structural parameters f 


(ù) 2 >y- 1 +p. 


9D E Lus uude re > 25 dioit ^ go, 1, 
|» D3hndga-im 0, 


(0) hold for a structure of the 
« and g;;, are locally identifiable. 


laten 


, 


b» f.91..93.. 555 gs. > 0. 
CO: E hausse. А 


25 аан «qm. 
аге continuous funetions of Йе s Bue +e y Opre and possess continuous first 
and second partial derivatives. 
(v) At least Y — 1+ py of the expressions 


Lo Ldadós 080 gy ; 


е , 
Кыр, чар ; 


are functionally independent. 
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This theorem represents an application of a more general proposition, 
the heuristic origin for which is to be found in a classic paper by Fisher (3) 
on the limiting form of the Karl Pearson chi square criterion. A rigorous 
statement and proof of this more general proposition has been provided by 
Neyman (8, p. 250) in connection with the property of consistency of point 
estimators, i.e., the property of convergence in probability of an estimator 
to its corresponding parameter. That the proposition arises in connection 
with this property of estimators is not surprising since in ordinary practice 
only sample, not population, data are available. The following paragraph 
is concerned with the feasibility of the hypotheses (7) through (v) of Theorem 
1 for the latent class model. This theorem is then related to the point estima- 
tion property of consistency. 

Hypothesis (7) is equivalent to (3.1) and can be readily checked in any 
particular application. For example, it is satisfied in the example in section 6, 
where р = 4 and y = 2, so that 


2? =16>2-1+ 00) = 9. 


Hypothesis (ii) follows directly from (2.2) and (2.3). Specification (iii), on 
the other hand, cannot be verified directly; however, substantive considera- 
tions may strongly suggest it. For example, for p = 4 and y = 2, if the con- 
trary were true and 


7» f[.91: 02:503: 4:2 = 0, 
PES 


then either f, = 1 (so f; = 0) and an even number of the g;, = 0, or an 
odd number of the g;;. = 0—implications that are highly unlikely from the 
psychologist’s viewpoint. (Thus, f; = 1 implies а degenerate application of 
the latent class model, viz., that almost all subjects belong in a single latent 
class—a vacuous scaling of £. Again, g;» = 0 implies that the 7th subtest or 
item has almost no discriminating power and so is unlikely to be incorporated 
into the test initially.) The continuity hypothesis, specification (i), clearly 
holds since the functions are polynomials in Je and gi. Finally, specification 
(v) is equivalent to asserting that the rank of the Jacobian matrix of the 
expressions is y — 1 + pv. It follows readily that full rank for the Jacobian 
is in turn equivalent to the requirement that the information matrix I (de- 
fined in section 6) be non-singular. This condition on I may be examined by 
direct algebra for a given model, or indirectly by numerical calculation о 
the approximate inverse of Т, as in the example in section 6. 

The connection between local identification of a structural A 
апа the property of consistency is as follows: If it is possible to show that s 
estimator of the structural parameter 6, est. 0, is consistent, then 8 эш 
locally identifiable. For example, in the latent class model if it is d s 
to show that the structural parameter f. has an estimator est. fe whic 
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converges stochastically to f. , then fe must be locally identifiable. ba 
connection may be established by noting that if a structural Morum m 
not locally identifiable, i.e., in a neighborhood of the parameter, the mie 
is not uniquely determined from the probability distribution of the observ s 
variables, then every estimator of this parameter will fail to converge 1 
probability to the parameter. nM А ' 
In terms of estimators, the application of the Neyman general propost- 
tion (8) to the latent class model means that Theorem 1 may be stated as 
follows: If the specifications (7) through (v) hold for a structure of the latent 
class model, then the estimators f. and ĝ;.. of (4.2) are consistent, i.e., the 


set of solutions f. and ў,,. of (4.2) are such that f. converges in probability 
to f. and ф; to g;,. as n tends to infinity. 


4. Efficient Estimation of the Structural Parameters f. and g;;. 


From the p-dimensional binomial distribution 
Ga, , 2» , +++ , £) and from (2.2) 
the structural parameters f. and g; 


of test responses 
it follows that the likelihood function of 
< based on a sample of n examinees, 18 
L(f. ? gii) = (52 f. selze cis бл)" M бї ей. мы а fa (4.1) 
кл (> FGF eGBic йг oe, 


where 7;,;,...,, is the number of subjects in the sample who give the response 
pattern 7, 4, ==- 7 


Application to this likelihood function of the method of maximum 
likelihood yields f. and Îi: as estimators of f. and бф . These estimators 
are the solutions of the set of equations obtained by differentiating the loga- 
rithm of (4.1), viz.: 


or раан, " 
3 (4. 
"o9 500.0 ual LE TELE MEN 


Not all applications of thi 
estimators which possess the pr 
give examples]. However, as n 
the specifications (i 


eyman and Scott (9) 
and ф;„ are consistent if 


‚ for examples]. 


THEOREM 2. If specifications (2)- (v) 
of the latent class model, then the estimators 
That is, f. and ў. 


of Theorem 


Ía and ĝ 


have a joint asymptotic Gaussia 


l hold for a structure 
ise ате asymtotically efficient. 
n distribution, and any other 


—— NY — = „> —-—8—ө—=є—_—————. 
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estimators of f. and g;,, which are consistent and asymptotically Gaussian 
have asymptotic variances exceeding the asymptotic variances of f. and $i; > 


[Proof of the general proposition, of which Theorem 2 is a special case, can 
be found in Neyman (8, p. 250).] 

As is the ease in most applications of maximum likelihood, the solutions 
f. and 6... to the likelihood equations (4.2) are not in general expressible in 
closed form. The indirect approach of iteration must therefore be employed. 
That is, a trial solution f, and g;,. must be assumed and a linear system 
solved for small, additive corrections ôf, and 80; , after which (6.1) is applied. 
A convenient mechanization of this procedure, due to В. A. Fisher, is called 
the scoring system. The technical details of the scoring system are given by 
Rao (10). A. numerical example is given in section 6 to illustrate the com- 
putational procedure. 


5. A Statistical Test for the Number of Significant Latent Classes. 


The final problem of statistical inference considered here is that of 
formulating a statistical test of the hypothesis that y = Yo , where yo denotes 
a specified number of latent classes. 

A natural approach to this problem is direct examination of the goodness 
of fit of the manifest marginal probabilities g;,;,...:, ; generated by the struc- 
ture via (2.2), to the actual population manifest marginal parameters g?;,...;, + 
Thus, if the discrepancy between the generated G(x, , 1, +++ , z;) and the 
actual G*(v, , 2, , ++- , х„) is substantial, the psychologist might postulate 
Y = yə + lin an effort to improve the fit. In practice, however, direct exam- 
ination is generally impossible; statistical inference is required since only 
the sample estimates т;,:,...:,/% not the actual population parameters 
0*5,:,...,, are available. Here n;,;,...;, 18 the sample observed response fre- 
quency corresponding to subtest response pattern 42, -:7, (ef. Table 2). 

For this formulation, the classical chi square goodness-of-fit test would 
be applicable if the structure were completely specified independently of 
the set of sample data (n; ios]. Thus, for a postulated уо and known 
f. and g;,. , the parameters 0%, #,..-# of G*(z, , ж» , *** , %) would be deter- 
mined by (2.2). Hence the discrepancy between observed and theoretical 
frequencies, i.e., between the estimates ni, iai» of ng ;,...;, and the туё г... 
as calculated from (2.2), could be tested by: 

x = 29 Vn ~ ngisi) NPR iain» 
D 
which is distributed approximately as chi square with degrees of freedom 
2° — 1. 

However, in practice the structural parameters f. and g;,,—and there- 
fore g*;,...;, aS obtained from (2.2)—must be estimated from the sample, 
hence the classical chi square test above is not valid without modification. 
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THEOREM 3. If specifications (i)-(v) of Theorem 1 hold for a structure 
of the latent class model, then the following quantity in the limit as n tends to 
infinity is distributed as chi square: 


xm а “Яңа, Ад aser (5.1) 


with degrees of freedom 2? — «(p + 1), i.e., degrees of freedom. equal to the 
mumber of independent. manifest paramelers g;,;,...;, minus the number of 
structural parameters ў. and g;,. . Here fisia., 18 the sample latent response 
frequency corresponding to subtest response pattern йй +4, 5 he, Dus 
is generated by the structure upon replacement of the stru: 
by the efficient estimates, e.g., for p = 4 and y= 


d 
ctural parameters 
2 as in section 6, 


thos = ДЛТ ЛЛК zi 526::202:03:204;3] 


Afrigorous generalization of this proposition is given by Neyman (8). 


6. Illustrative Example 


In order to illustrate the calculations needed, a latent class structure 


is estimated from data obtained by Schumacher, Maxson, and Martinek 
(12). Four machine design subtests, given to 137 engineers, were dicho- 


tomized into positive, 1, (above the subtest mean) and negative, 0, (below 


TABLE 2 


Frequency of Occurrence of Response Patterns 
for the Four Machine Design Subtests (12) 
chine Design Subtests (12) — 
Response patterns Observed Generated 


Frequencies Frequencies 
КО а 


hog DT. 

125g ELA iiid 
1234 23 18.276211 
T234 8 8.449612 
1234 6 6.882195 
1234 5 8.915412 
1233 5 8.789920 
I234 9 3.676943 
Г254 3 5. 386429 
T234 2 4.545934 
1234 2 4.123700 
1234 3 3.602141 
1233 14 5.034202 
[234 8 8.185339 
I234 3 4.205489 
T2354 8 8.586886 
1234 4 5.689062 
T233 34 32.650525 


the subtest mean). Table 2, second column 
Таз, With which the observed response pa 
To apply the latent class model, som 


; gives the sample frequencies 
tterns 1112137, occurred. 


е hypothesis concerning ү, the 


, 
| 


а 
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number of latent classes must be made. Subsequently, this hypothesis is 
tested to determine whether the number is in fact sufficient to account for 
the data. In this example, the hypothesis is: y = 2, i.e., that two latent 
classes (Creative versus Non-creative on machine design) are sufficient to 
account for the observed response pattern frequencies. 

To start the iterative scoring system of Fisher (10) it is necessary to 
find first approximations f. and g;,. to the structural parameters f. and 0;;: . 
Then efficient estimators f. and ў;,. are given to a first iteration by 


Һе + of; (6.1) 
ji: = fu “Р 80... , 


where ôf, and ôĝ:; are the corrections to be added to the trial solutions. 
Moreover, one iteration of the scoring system is sufficient to yield estimators 
f. and fiic which are fully efficient, provided the trial solutions f. and 9i. 
are consistent (8, p. 255). Since the Anderson-Lazarsfeld-Dudman estimators 
(1) have this property, they are used in the present example as the first 
approximation (Table 3) to the structure. 


TABLE 3 


First Approximation [ Anderson, (1) Z to an Efficient 
Estimate of the Latent Class Structure 


Machine design ability t 
Latent classes (domain off ) 1 ("creative") 2 ("non-creative") 


Latent marginal probabilities T| 2.516228 1; 2.483772 


Subtests x; Subtest conditional probabilities 


|; 7.729419 1,2 = 080928 
$1, 7.270581 ®т;; =:919072 
Ez; = + 745642 2,2 = + 188564 
©з; 7.254358 95,2 =:811436 
$5, 7.174096 85,57. 112673 
gy =: 225904 4s, .887327 
Жу Fe ABTS "Шр 2028393 — 
~ 


7 
S 
у 
@ 
> 
5 
m 
f 
" 
S 
i 
2 
& 
5 
5 


m 


For the latent class structure postulated to underlie the data of Table 2, 
an efficient estimate is given in Table 4. Asan example of Table 4, Ji = .581917 
because from Table 3, f, = .516228 and, by the method of caleulation ex- 
plained below, ёў, = .065689; hence, using (6.1), 


fir = Ji + Of; = -516228 F 065689 = 581917. 
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TABLE 4 


An Efficient Estimate of the Latent Class Structure 


Machine design ability Е 
Latent classes (domain of $ ) 1 ("creative") — 2("non-creative") 
Latent marginal probabilities f, -.581917 f, =. 418083 


Domain of L^ 


Subtests x; 


Subtest conditional probabilities 


81, 7.686730. g] =. 114614 


s ў 
1 zi 

8T;1=.313270 8j,2 7.885386 
728419 g . 173071 

ы 
‚271581 ‚826929 
-676457 078556 

x 
3 323543 gy, = 921444 
M 84,1 7-676904. в, „= .173385 
83, 7.323096 gi, =.826615 


Finally, the corrections 5 f. and 60, 


аге obtained by the scoring system 
from the relations: 


1S Tocog DM FOLTOS 
of, =- Z nl Rens y P. MI O 


$-dboc't-l 


(c = E se Пр Жашы 1); 


(6.2) 
LS wy 1 Sy 25 « 
80... == У) nt ue o xy NTO eR 
N erar N iTar etry TUNE 
(i = 1 =.. VB ES 1. M „Ӯ. 
Here {һе tilde on the S and the I indicate evaluation at the trial solutions 
J. and ў. of: 


(i) the S functions, one for each structural parameter, which are defined 
as 


9(log Т; 
s, = 008) @=1,...,у—1у, 
(6.3) 
E n 9(log L) 


Әй 1 = 1 


2S оис = Дуза. уу, 
ie., simply а relabeling of the left 
to as the efficient scores for the para 


the efficient scores as evaluated at 


side of (4.2). These functions are referred 
meters f. and g;... For the data of Table 2, 


—-—— 


" A 27. 
= ee о — A 
ee алг... 
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TABLE 5 


Efficient Scores S, and $,.. Evaluated 


at the First Approximation 


51 = 10.635378 


5р, 73.624402 Si. = 35. 114865 
Sp.) = 1. 260259 6.861212 
= = 

.| = -25.12009*- -5. 782032 


-16.631863 $ -9. 284267 


2 


TABLE 6(a) 


Information Matrix per Single Observation, 


Evaluated at the First Approximation 


" Bi; ud 83:2 Bau 
1| f, 3.048439 0.509498 ... 0.678498 0.365129 
2 Bu 0.509498 2.017739 ... -0.526254 -0.255081 
3 82.) 0. 354050 -0.124733 ... -0.392939 -0. 173048 
0.365129 -0.255081 ... -0.127786 2.136945 


TABLE 6(b) 


Inverse of Information Matrix per Single Observation, 


Evaluated at the First Approximation 
————_—_S— EE Pm 


1 ЫН! 832 542 
1 fi 0.588484 0.254609 ... -0.211503 -0.183881 
2 {В1;1 -0. 254609 0.677139 ... 0.202876 0.157569 
3 -0.187074 0.118647 ..- 0.141973 0. 109468 


82;1 


84,2 -0-183881 0.157569 ... 0.093239 0.561545 
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parameters pairwise, the information functions, expressed per single observa- 


tion, are: 


1 1 
a loch ES п 205.6.) (с,с' = 1, Y 1), 


1 
nen = EEG So ur 


1 1 
glenn = 7 BSS...) 


€ =r 
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i.e., essentially the variances and covariances of the efficient scores. The 
square matrix of these elements, of order y — 1 + py, is denoted by I and 
is called the information matrix. Hence in (6.2), п“, п1°7 *?, and 
nl99''*? constitute the elements of the inverse of the information 
matrix, after evaluation at the trial solutions. For the present machine 
design data, the matrix (1/n)1 is given in Table 6(a) and nJ~ in Table 6(b). 
Explicit formulas for the calculation of (6.3) and (6.4) are given in the Appen: 
dix. 
As an example of the use of (6.2), 


5f. 5 : nS, si nog a + nTVens, СЕСЕ nT 4 5,2] 


1 
137 [(.588484)(10.635378) + (— .254609)(3.624402) 


+ (—.187074)(1.260259) + -.. + (— .183881)(— 9.284267) = .065689, 
using Table 5 and Table 6(b). 

From (6.2) the crucial step in the scoring system is the inversion of the 
information matrix, ie., the existence of I^! is a necessary condition for 
identifiability. That the non-singular character of I is also a sufficient con- 
dition has been noted in section 3. 

_ Since I” is the asymptotic variance-covariance matrix of the efficient 
estimators of the structural parameters, it is often desired to obtain the 
first. approximation inverse matrix 1! explicitly. [As an example of the use 
of 1^, from Table 6(b), the variance of ЕЕ approximately V(fi) = 
-588484/137 = .00429]. However, it. is worth noting that if estimates of 
the large sample variances and covariances are not sought, then (6.2) is 
equivalent to solving a set of simultaneous linear equations in the corrections 
ôf. and [M without explicitly inverting the matrix 7. That is, as a matrix 
equation (6.2) is § = 1-'5, which is equivalent to 7 = S, Where 


5}, 8, 
rud hs 


89534 Say * 
For the machine design data (5.1) is i 
, А applied to Tab i to 
test for the number of significant late classes, УЮЫ E 


х? = (23 — 18.276211)?/18.276211 Te 


+ (34 — 82.650525)?/32.650525 = 33.01 
with 2' — 2(4 + 1) = 6 degrees of freedom. Since the probability of a chi 
square this large by pure chance is less than -001, there may well be justifica- 


ЕЗ 
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tion, with respect to the universe of examinees and psychological test used 
(12), for rejecting the hypothesis of y — 2 latent classes for the latent class 
model and for assuming the existence of more than two classes. 


Appendix 
Evaluation of the S and 1 Functions for Tables 5 and 6(a). 


(i) The efficient scores S of Table 5. 


Differentiation of the logarithm of (4.1) and evaluation at the first 
approximation leads to the general computation formulas 


S. = Эб (cy сбу ба тебуу басе +» › 
Е (1) 


S. = У) fuio tiii + 
D 


Thus, the efficient scores are linear functions of the response frequencies 
Жаа» , With coefficients q which are functions of f. and $i , viz. the 
quotients: 1 
cth term of (0:,:.:-:/7) 

= , 


a 


Üoytda i = 
nS (2) 


cth term of (fi, d), . 


(io ciiin mand 
That is, the denominator of a coefficient 0 is that manifest marginal 09:,#.--:#» 
indicated by the second subscript on q, and evaluated at the trial solution. 
In order to obtain the numerator of 4 readily, it is convenient in the deter- 
mination of the denominator to compute each of the y terms separately 
(later adding them, of course, to get the complete denominator). Then as 
(2) shows, the numerator is easily obtained in one operation, viz., by dividing 
the cth term of the denominator by f. (if dealing with S.) or by gs; Gf dealing 
with S;,.). 
For example, from (1), for p = 4 and y = 2, 


S. T (cim 123491234 + @‹зэ)у‹(ї?за)?Її2з4 + eee + doinassoftizss › 


which, for the data of section 6, is = — 5.782032 as shown in Table 5. For 
the first coefficient, Q(s;20230 › the manifest marginal for the denominator 
İS 01234, corresponding to the second subscript 1234 on d, which from (2.2) is 


= fifiagsngsungaa A fogs :292:293i294:2 + 


91234 


Hence, the denominator is obtained as the sum of the following two parts 
(c—1,2-— т) Ў адгадзада апа 70 :202:20з:204:2 . The numerator of 


(m doa) 18 obtained by dividing ds; into the c — 2nd term of the denomina- 
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tor, foGr.292:2Ga:2Gs:2 , giving Jaĝı:22:2014;2 . Hence by (2) the first coefficient is 
э 291; 3293524452 


Тоба ss Gace " 
Аф адзафзадка + fedi айз эйз 2942 


(a:masse = 


which can be caleulated by direct substitution from Table 3 to be .010358. 

'The other coefficients are obtained in a similar manner. А 
Actually, for a given efficient score S, only half of the numerators of ve 

coefficients d need be calculated. For example, out of 16 numerators for 


©з» only 8 of the numerators need be computed, each of these being the 
negative of one of the remaining 8, eg, Jj 


:292:29s;2 , Which occurs in 
Qc: азза) , is the negative of — 0,0 й, : 


which occurs in (3,2) (1234) + 
(@) The information functions, T, of Table 6(a). 


Evaluation of the expectations (6.4) at the first approximation leads to 
the general computational formulas: 


1+ {numerator of (cc. [numerator of (coo. 
moes У, = 


Jis. 


E| р 


1 


I [numerator of ў,;..у(:,:,...; »][numerator of d ciues] 
"p x Ras - , 
(8) 


17 {numerator of fenti.. »][numerator of ЕТЕУ } 
n (I)e) = 2 ret 
D 


giu 

"Thus, comparing (3) with (2), 
from the calculations already 
For example, from (3) an 


one notes that Table 6(a) can be obtained 
made in completing Table 5. 
d (2), for p = 4andy = 2, 


- Tan = (086 term of Gr2s)/G2:1][(2nd term of G1234)/Ga:2] 


боза 


+... 4 [st term of 91333) /Go:1)[(2nd term of бтз) /Gs:2] , 
izz 
which for the data of secti 


on 6 is —.392939. Specifically, the first of these 
terms for Ут) Enma is 


а азад эй» ag. a] 
Аф адаф + DN ; 
which сап be computed directly from the 


à caleulations previously made for 
aina ОЁ 5%, [as illustrated in (0 above] and for femaza Of за › 
which can be similarly illustrated. 


л 


on 


12. 


13. 
14, 
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THE SELECTION OF JUDGES FOR PREFERENCE TESTING* 


R. DannELL Bock 
UNIVERSITY OF CHICAGO 


A scheme for choosing a few individuals whose preferences for given 
objects are most representative of those of a larger group of individuals is 
proposed. The method involves (a) quantifying the preferences of each 
individual so as to discriminate optimally among objects, (b) testing statis- 
tically whether or not a common preference continuum may be assumed for 
the quantified p (c) constructing a linear estimator of values for 
the objects on this continuum, if it may be assumed, and (d) selecting as judges 
the least number of individuals whose quantified preferences, when used with 
this estimator, determine values for the objects with acceptable accuracy. 
A numerical example based on food preferences is presented. 


I. Introduction 


Preference studies, particularly of foods, frequently depend upon a 
limited number of judges who have been chosen from a larger group of 
individuals. Common practice is to choose those individuals whose repeated 
preferences for the same objects are most reliable. This assumes that all 
individuals reflect a common dimension of taste and that an individual 
Whose preferences are reliable, but atypical, will not be encountered. A more 
thorough method would include a test for dimensionality and, if this con- 
dition is met, the selection of judges who are most representative of the group 


аз a whole. 
Let us assume that the preference of the ith individual for the jth object 


on the kth occasion is determined by a preference score g;;, , which has the 
composition 


0и = QiWi Tes, { (1) 


Where o; is а coefficient characterizing the ith individual, w; is a value charac- 
terizing the jth object, and e;;, is a random error distributed as NO, 3) 
and independently of all o; , w; , and other б. As usual, the variance of 
9; jx over all occasions and objects is unity and the variance of w; over all objects 


is taken so that 
аот (2) 


supported in part by the Quartermaster Food 
Forces. Views or conclusions expressed herein 
rily reflect the views or indorsement of the 


*Preparation of this paper has been 
and Container Institute for the Armed 
are those of the author and do not necessa 
Department of Defense. 
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The problem is to test the compatability of the observed preferences with 
(1) and to derive a scheme for estimating the ш; which will most efficiently 
utilize à limited number of judges. 

The preference scores are not observable, of course, but are presumed 
to underlie ratings, rankings, or paired comparisons of the objects, and are 
assumed to be estimable only by resort to a scaling method which quantifies 
the preferences. In the present context, an efficient and practical method is 
to choose scale values for the preferences which will yield a minimal estimate 
of ¢; . This type of scaling has been proposed by Fisher (6), in a different 
connection by Guttman (7, 8), and by Maung (16), Johnson (12), and Bart- 
lett (4). Related tests of significance have been discussed by Bartlett (2), 
Fisher (5, 6), Williams (19), Marriott (15), and others. A suitable version 
of the method for the present purpose is as follows. 


II. Scaling the Preferences 


Objects 
1 2 = q 
1 ty te lia 
21] da taz log 
Occasions : 5 
ty; 
p br і іа (8) 
1| ва n 21 па | n.. 
А 2 | т. N.22 Nig2 | Wo 
P | Qa, т, fies | dos 
e c id рук" 
Nil No, Ma. In... 
à "ra We а эы of preference on the kth occasion for the jth object 
= 1,2,---,7”): 


(a) For ratings, t indicates a quasi: anti i as 
TY -quantitative y ood 
еч 7 categor such g $ 
(b) For rankings, t indicates а position in an 


(c) For paired comparisons, / indicates the number of times one object 
is preferred to all others. 


ordering. 


n,; is the frequency with which the /th category oceurs for the jth object 
over the p occasions. 
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т... is the frequency of the /th category for all objects. 


п.;. = Р, 
п... = 0р = "п, 
r € 0. 


Representing the estimated score of the /th category by £: , the average 
preference score for the jth object on p repetitions is defined by 


> 20. за D UN je 
n 7 


Te 4 
gi. EP Ec (4) 


where a subscript for the ith individual is tacitly understood. 

Since interest in preference testing is primarily in distinguishing quanti- 
tatively between objects, scores should be assigned to the categories which 
Will discriminate optimally among the objects, where optimum discrimination 
is attained when the sum of squares between objects is a maximum with 
respect to that within objects. But for sums of squares: 


Between + Within = Total, 


So this is equivalent to maximizing the between sum of squares subject to 
the condition that the total sum of squares remains constant. That is, assign 
values to z, which maximize 


LO (Lam (ат), 6) 
subject to the condition that 
p» TNI — (> EM.) /n (6) 
n t 


be a finite constant. u 
Since this maximum is independent of the origin and scale of the =, , 
impose the conditions that scores for the sample sum to zero and have unit 


variance. That is, 


» xm. = 0, (7) 


and 


(8) 


| 
3 


2; an. = 
Accordingly, (5) and (6) become 
by (55 хт.) /®› (9) 
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and 
БЕ xh... (10) 
7 

To express the required maximum explicitly as a function of the ob- 
served frequencies, it is convenient to adopt matrix notation and to define: 

T= [m ta +++ Ж +++ 2,] = row vector of assigned scores; 

ЁК=[п]=тх q matrix of frequencies in (3); 

FF'/p = H; 

D-—[n]-rxr diagonal matrix of marginal frequencies in (3). 
Then (9) becomes 


x 9» zm.) /p = Hx’, (11) 
and (10) becomes 
Livin... = spy, (12) 


The maximum of tHz’, subject to the condition that zDa' = n, is 
obtained by differentiating and equating the resulting expression to zero: 
tHe’ — "(Da — n), 
Where 7” is a Lagrangian multiplier. The result is 


(Н — рр) = 9, (28) 
Which may be Written 
"HD" — pog s 
For z to be non-null it is Decessary that the determinant 
japa tar 0; (15) 


where T іѕ а Toot of (15) and z is the Corresponding latent vector of НЮ”. 
An iterative methodi [cf. Hartree (10, p. 178f.)] may be used to obtain 
tr E a of HD^' with arbitrary origin and unit. The first of these 
vectors, wi т = Land z = (1 5-91) ig ы: 7 ided by 
à E : ; 18 Wivial and сап be avoide 
replacing F with the correspon rix of deviations EO expectation. 
ire 
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Transposing and post-multiplying by Hs 
г„Нг = 222022 = тп. (16) 


Similarly {°, which estimates ¢°, is the fraction of sum of squares within 
objects, and 


nee = is (17) 
To transform z, into z, impose conditions (7) and (8) to obtain 
x, = uz, — v), (18) 
where 
v= doam,../n, (19) 
and i 


= EN. "we (20) 
V o (e v) Re 


In general HD^' has r — 1 non-trivial latent vectors, of which the 
first, x, , provides estimates of the maximally discriminating scores for the 
preference categories, In the present context further interpretation for 
any remaining vector, even if its associated root could be shown significant, 
does not seem necessary, although it should be noted that relevant inter- 
pretations are available (9). 

The statistical significance of the scores may be questioned on two 
counts: If they are to be useful for discriminating among the objects, variance 
attributable to differences in column means (mean preference scores) must 
'be significant. Fisher (6) suggests that an approximate test, based on an 
analysis of variance, be made directly from the value of 72 as shown in Table 1. 


TABLE 1 


Form of Analysis of Variance for Derived Scores 


Source of Degrees of Sums of Mean 
variation freedom squares squares F 
E iq(p—1)—(r—D 
Between Un 290 x 
objects d obse тп 417—2 (1—2) (a--r— 2) 
a j д п(1— т) 
esidual q(p— 1)—(r— 1 n—nm 2920260 = 7) 


"Total n—1 n 
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[Cf. Williams (19).] The degrees of freedom are adjusted for the шу 
constants fitted by adding т — 1 to those between objects and subtrac is 
r — 1 from those of the residual. The scores may be considered useful only 
if F is significant. a 

мй Te is also of interest to test whether these derived, maximally diserimin- 
ating scores are any real improvement on arbitrarily assigned scores, e.£., 
1,2, 8, +++ ‚т. Again following Fisher (6), this may be done by testing the 
significance of variance between objects, computed from the derived scores, 
which remains after variance attributable to the assigned scores has been 
removed. The appropriate analysis of covariance is shown in Table 2. (The 


TABLE 2 
Form of Analysis of Covariance of Assigned and Derived Scores 


Source of Sums of squares Sums of cross- Sums of squares 
variation (assigned scores) products (derived scores) 
Between objects ЕНЕ ЕНІ zHz' 
Residual D} — tHe’ Dz’ — EHz' zDz' — zHz' 
Total ЕРЕ £Daz^ хрх 
vector of arbitrarily assigned scores, must satisfy the relation У), gn., = 0.) 


The analysis of variance of the derived scores, eliminating the assigned 
Scores, is shown in Table 3. The degrees of freedom between objects has been 
reduced by 1, since the elimination of £ leaves only r — 2 scores adjustable. 


TABLE 3 


Form of Analysis of Variance of Derived Scores, Eliminating Assigned Scores 


Source of Degrees of 


гсе Sums of Mean 

variation freedom Squares Squares F 

Between 4g 88, : Obtained by 88, S8,q(p— 1) —(r—1) 
objects f subtraction 447—3 ШЕ Сортан 


SS, = {хрх — Hx 
Residual 900—1) | 


88, 
aio o 04) 9—1) 01) 
t= { EDE ЕНЕ 


` 2 
Total n—2 88, =: TE 


NN 
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А significant value of Ё is necessary, of course, if the derived scores are to be 
preferred to more convenient assigned scores. As before, this test is only 
approximate. For a more exact treatment, see Bartlett (4). 


III. Estimating the w; 
If written explicitly, the scaled preferences of N individuals for g objects 
on p occasions would appear as in Table 4. For subsequent computations 


TABLE 4 
Preference Scores and Mean Preference Scores 


Objects 
1 2 РЗ q 
Individuals Individuals exi. Individuals 

Occasions 1 2: Ñ i & e N Le i 2. N 

1 gii goi *** gnu Jizni J221 ^^^ Nn кэе fii 23 ^7 : Оха 

2 fis 9212 ccc (wi 9122 J222 ^^ 7 N22 STA Qie2 #2 ^'^ Noz 

[4 Qu» фә) ccc (Nis Фәр J22» 7C 92р Jian зар *** INav 
Means Qn. 921. *°* m. gi». 922. "7C 9№2. Jig. Jaa. "C7 ма. 


only the mean preference scores are needed and may be obtained by the 


relation 

gii. = xni;/D, (21) 
tent vector for the ¿th individual under the 
h column of the corresponding F matrix. 
from the preference scores of the N in- 
equirement of optimal discrim- 
timator a linear compound of 
f squares of the combined 
sum of squares of the 
on the kth occasion be 


(22) 


where 2, is the first non-trivial la 
transformation (18), and nz; is the Jt 
A method of estimating the в; 
dividuals is needed. Resorting again to the r 
ination among the objects, choose as an es 
the preference scores which maximizes the sum 0 
scores between objects on condition that the total 
combined scores is constant. Let the estimate of w; 
Wik = 2 YiGiik » 
and for all occasions 
Aj. = > YiGii. * (28) 
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The analysis of variance of the preference scores with respect to this 
linear compound may be represented as: 


SS DF 

Between Objects yMy' q-—1 
Within Objects yBy' q(p — 1) 
Total yTy' gp 


where M, B, and T are, respectively, matrices of between objects, within 
objects, and total sums of squares and cross products for the preference 
scores as represented in Table 4. Only M must be computed explicitly, with 
M = pGG', (24) 

where G is an N X q matrix of mean preference scores (9:;.] 
The coefficients of the required optimal 
pound are the elements of y determined from 


y(M — dT) = 0, (25) 


ly discriminating linear com- 


where А is the largest root of 


[М — AT | =0. (26) 


justified. In this case the о 


ff-diagonal elements of 
equal and 


T and M are asymptotically 
B-T—M 
approximates an N X N diagonal matrix with elements 
ba = x Уйа р D gh. . 


By (8) and (16) this becomes 


b =n – (pz, F Fiat) /p? 


=n mă; 
and by (17) 
bis = ng. (27) 
Taking 7 = | аз ап approximation of В, 


| M -NM + Z) | =0, 
or, letting ы 


и = т\/(1 — у), (28) 


ln <В. а 


us 1 2 í Jue сы 
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with m = q(p — 1) — (r — 1), the additional r — 1 degrees of freedom being 
removed because of the constants fitted in sealing, 


| MZ! — Ш/т | = 0. А (29) 
Taking the largest root of (29), y may be determined from 
yMZ' — pI/m) = 0. (30) 


[Cf. Bartlett (3).] 

If m is clearly significant, a basis for testing of the significance of varia- 
tion attributable to the remaining roots is provided by Bartlett’s (2) approxi- 
mation for Wilks’ ratio 


min(N.c-1) 


x--[-iN-cg] X 108. (1 — м). (31) 
=1 
For large n this becomes approximately 
min(N.a—-1) 
"e ae (32) 


a-1 


That is, the latent vectors associated with each of the canonical variances 
transform MZ ' into min(N, q — 1) asymptotically independent quadratic 
forms, each distributed as x^ with degrees of freedom equal to the number 
of arbitrary constants fitted. Specifically, the sum of the canonical variances, 


m-tr (М27) = ш + р ЕБЕР Еа 3 y (33) 


is distributed as the sum of min(N, q — 1) asymptotically independent 
chi squares with degrees of freedom 


№ = 1) = (у +4—2@+(%0+а-—4®+ 
+ {N — g — 2[min (N, q — 1)]}. 


'То test the residual variance, eliminate that of the first root by taking 


x = mr (MZ) — Шш (34) 


on N(q — 1) — (N + — 2) degrees of freedom. 
ied aS has in general min(N, q — 1) roots, ш, үл T ү m А 
when ordered by size, giving the canonical variances of oe z 3 ( D: olds, 
only и, will be significant and the linear RUDI вс Un уй! ы hp 
responding latent vector yı will account for all the uoce variatio ш 
{һе preference scores. In critical cases (31) would be somew at more accurate. 
It is of interest to note that if (1) holds and p is large, 


gii = 050i 
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approximately, and 
G = ао. 
If 
cows’ /q =1, 
GG'/q = аа = Mp. 
Then (30) becomes approximately 
y(na'a — uB/n) = 0, 
where 
| пола — uB/n | = 0 
has only one root: 


и = naB w, (35) 
and one associated latent vector: 
= -1 
y — naB (36) 
25 naZ, 


a result identical in form with that obtained when constructing a linear 
diseriminant function for two g 


similar also to that for the estimation of mental factors [cf. Holzinger and 
Harman (11, p. 322)]. Р urthermore, from (2) and (17) 


2 2 
о = т, 
and, asymptotically, 
2 = [ni], 
or, by (36) 
y; = л/(1— 7). (37) 


between a fallible Score and a theoretical 
"true" score, then 


and 
y; = Vr4/(1— tu). 
IV. Selecting the J udges 


In the sample the best estimator of c; is given by the first latent vector 
y associated with the solution of (30). Individuals for whom the corresponding 
elements of y are larger are to be preferred as judges. If k such individuals 


roups [cf. Kendall (14, vol. II, p. 341)] and à 
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are selected, the question arises whether the loss of information caused by 
excluding the preferences of the remaining N — k individuals from the 
estimates of the w; is appreciable. 

If y, is the vector у after the elements for the unwanted individuals 
have been dropped, and M, and Z;' are M and 27! with corresponding 
N — k rows and columns dropped, then adjusting y; so that ушук = 1, the 
portion of the canonical variance ди, which is attributable to the k selected 
individuals is 

v, = my,M,Z; yt - (38) 


The significance of the remaining variance can be tested approximately by 
taking 
Hı — Vk (39) 


as x? with (N — k)(q — 1) degrees of freedom (cf. Rao (18, p. 257)]. If this 

variance is insignificant, or appropriately small with respect to ш, , the 

каз of the k selected judges can be considered adequate for estimating 
ewj. 

Since the preferences within objects are uncorrelated, the elements 
remaining in y, are unchanged except in scale. If it is convenient to have 
standarized values for the w;, within objects and between occasions, the 
elements of y, should be multiplied by 


Е -1/2 
С ЕР БЕТА T4 M . (40) 


V. Related Problems 


It should be understood that the sample dealt with here is, in effect, a 
series of occasions out of infinitely many in which the preferences of the same 
individuals might be repeated under the same conditions. Predictions made 
on the basis of this sample apply only to future preferences of these individuals 
and are not immediately generalizable to the population from which the 
individuals are drawn. When a single significant dimension of taste is found, 
however, this limitation is less serious because there is no evidence which 
contradicts the assumption that a new group from the population will share 
the same dimension of taste and show the same relative preferences for the 
objects. When significant additional dimensions are found, the problem 
becomes much less tractable but jn some ways more interesting. Additional 
dimensions raise the question of what and how many attributes in the objects 
are being evaluated by the individuals and reflected in the preferences. 
This is essentially a problem in multidimensional psychophysics; the type 
of canonical analysis used in the present study appears to be an alternanve 
to approaching the problem by a direct reduction of a table of oer 
from paired comparisons of the objects. Preference scores for the additiona. 
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dimensions may be estimated using the other significant latent vectors ue 
(30), and rotation of these vectors to yield more meaningful scores a та 
missible. In practical applications, determining dimensionality am a 
important because it would indicate the minimum number of attribu е 
which must be controlled if the objects are to be produced in uniform quality Е 
А natural extension of this would be to identify the attributes physically or 
chemically and subject them to planned control and development. А 
The converse problem would be to identify and characterize by inde- 
pendent variables those individuals whose preferences are responsible for 
or most representative of the various dimensions of taste. If individuals with 
atypical tastes were few, they could be excluded from further preference 
testing and the remaining homogeneous group considered representative of 
the ultimate consumers of the objects. Alternatively, it might be found 
that distinet subgroups of the individuals with different tastes can be iden- 
tified in terms of sex, race, income-group, ete. A set of separate panels repre- 
senting each of these subgroups would, so to speak, “span the space of pref- 
erences” in the population. If the proportion of these groups in the population 
were known, the reaction of the total population to new objects could be 
predicted by a weighted combination of the preferences of the subgroups. 
Finally, the relationships among the preferences of the groups could be 
studied by a canonical analysis or а multivariate analysis of variance in 
which the "individuals" of this study are replaced by groups and “occasions” 


by individuals within groups. A problem of this sort based on a national food 
preference survey is now under study. 


VI. Numerical Example 


Oberman and Li (17) report ten replicate ratings (p) of ten individuals 
(N) for pastries baked from five different fats (4). Ratings were made on а 
five-point scale (7): 1 = not edible; 2 = poor; 3 


= fair; 4 = good; 5 = excel- 
lent. For the first individual, the ratings of the five pastries are shown in 
Table 5. 


The body of Table 5 is the matrix F; 


the frequencies on the right are 
the elements of the diagonal matrix D. Then: 


62 42 85 01 00 
42 60 34 04 00 


pH = FF’ = | 35 34 38 10 08 


Or 04 15109537 "15 
00 00 03 18 09 


Column sum: 140 140 120 70 30 


= EE ——————<_ rl II. 
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The rows and columns of pH must sum to pn.., . To form Н D`’, divide 
elements in the /th column of pH by pn... . It is convenient at the same time 
to remove the expectation by subtracting 1/r from each of the resulting 
elements. Designate the resulting matrix by T: 


.2429 .1000 0917 —.1857 — .2000 

-1000 .2286 .0833 —.1429 —.2000 

T = -0500 .0428 1167 —.0571 -—.1000 
—.1929 —.1714 -—.1167 .3286 -4000 
(—.2000 —.2000 —.1750 0571 -1000. 


Column sum: .0000 .0000 .0000 .0000 .0000 


Columns of T must sum to zero. [Note: x^ for F, with (q — 1) (r — 1) degrees 
of freedom is n(tr T).] j 

The first latent vector of Т is 2, , the first non-trivial latent vector of 
Нр”. To extract 2, from T, premultiply by a trial vector orthogonal to the 
trivial vector (1 1 1 1 1). The vector (1 1 0 —1 —1) is convenient. For these 
data, five iterations produce the following essentially accurate estimate: 


2, = (.8325 .7831 .6021 —.7721 —1.000), 


with associated root 73 = 7921. Using (18), (19), and (20), impose conditions 
(7) and (8) to obtain 


(.6292 .5514 .2604 — 1.8691 — 2.2260). 


Analysis of variance based on 73 shows significant discrimination between 
fats (Table 6). Analysis of covariance of assigned scores 1, 2, 3, 4, 5 [adjusted 
to = (1.42 0.42 —.58 — 1.58 —2.58) so that У), £ n... = 0] and the derived 
Scores is shown in Table 7. Analysis of variance of the derived scores, 
eliminating the assigned scores, shows that the derived scores discriminate 
between fats significantly better than the assigned scores (Table 8). 

Seales for the remaining individuals are constructed in the same way. 
"The correlation ratios and results of the significance tests for each individual 
are shown in Table 9. The mean preference scores of each individual for the 
pastries are shown in Table 10, the body of which comprises the matrix G. 
(Rows of G must sum to zero.) The elements of the diagonal matrix Z com- 
puted by (27) and (17) from the correlation ratios of Table 9 are: 


10.395, 32.690, 32.345, 34.885, 19.355, 13.795, 31.625, 3.365, 24.845, 9.265. 


The product MZ = pGG'Z-' is given in Table 11. 4 
The first latent vector of MZ ' із extracted by premultiplying by (1... 1) 


TABLE 5 


TABLE 7 


Analysis of Covariance of Assigned and Derived Scores 


Preferences of the First Individual 
Fats Source of Sums of squares Sums of cross- Sums of squares 
variation (assigned scores) ^ products (derived scores) 
Categories (t) 1 2 3 4 5 Total 
Between 44.08 41.01 39.60 
5 5-4 8 6 1 0 14 fats 
Residual 28.10 10.77 10.39 
4 # Em & p 14 
Total 72.18 51.78 49.99 
3 4 1 а а à 12 
2 $ b а д 6 7 
1 » o0 o в à 3 
Total 10 10 10 10 10 50 
TABLE 6 TABLE 8 
Analysis of Variance for Derived Scores Analysis of Variance of the Derived Scores, Eliminating the Assigned Scores 
Source of Degrees of Sums of Mean Solirce:of E s of Sums or E Mean т 
variation freedom squares squares F variation reedom squares squares 
Between 8 39.60 4.95 19.49 RESET. а m к m5 
fats , fats 
кк ds 20240 29+ Residual а 6.26 153 
Total 48 90300 psa Total 48 12.84 p«.0l 


£69V'VO7T oF теор 1966"  LLEUT 0/6 1618" 68512 OBL T 1026 8218" 1858' 
——=—= — LOs0't SZIO'I 80976 1029" VSO0'2 916C€l JEP 20667 LIS’ ZOL? 

to >d фут az tenprsoy 9vzs'€ OZZET 8I98'ET 968 $8)v'7 002/'1 210597 0569"  vi2S' ESOUE 
096L'7 1598 15862 0185 Sotel 691271 Ottt€ 6567 HIRT — 999v'2 


6t2€9901 £t DUNES ыс ыыы ы Сы 9001 LOPUT 609101 1108" 8292 110/1 &v8t 6ZLL 8819" 6809°% 
0Е59"Є 1Р80°1 06686 РІ" LOBE? EEBS'T BEZE 81017 885° EEEZ'E 
160@°1 (6229 6SbL°9 ti9€ 2126" 692" veer L162" зїї ӨТТ 
шорәәзл} 9LEB'Z viLL' $0899 LLYS’ €2l189'l OETI 60207 8626 919% LETS*Z 
á x jo saos89q gue ан S7€9'7 ЕР657 EEZT'S 96% 98091 1986 8011" 699% S676 0/27 
2 Leze"h VICUT 62656 BOTS  TólL'2 Socr TELE’ — 62109 6121" — S608'€ 

p jo ќацечотзчошта әчү ло} 1991 3 x 1- ZNO ONL 

71 ялуу uan 
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as the first trial vector. Four iterations yield an essentially stable first latent 
root (1/т) р = 26.0079 and first latent vector (with arbitrary unit): 


(2885 .0520 .0630 .0426 .1455 .2179 .0699 1.0000 .1038 .3276). 


The x^ test of variance remaining after that attributable to the first root 
has been removed shows that the preference scores cannot be considered 
unidimensional (Table 12). 

Inspection of the mean preference scores (Table 10) suggests that the 
departure from unidimensionality is а result of disagreement between the 
scores of the most reliable individual, no. 8, and those of the remaining 
more reliable individuals, nos. 1, 5, 6, and 10. Reliability is judged from 
the correlation ratios in Table 9. Individuals whose preferences show low 
reliability influence the test of dimensionality only slightly compared with 
those of high reliability. The disagreement is most m 
baked from fat no. 4. 

When results of this sort are encountered in practice, a careful review 
of the conditions of testing, the stability of the objects, and the training of 
the individuals is probably indicated. In the present example it would be 
particularly unfortunate to exclude individual no. 8, who is highly reliable, 
if through better controlled testing, retraining, ete., his preferences could be 
brought into line with those of other judges. 

On the other hand, a sin 


arked for pastries 


gle dimension in this example accounts for 
26.0079/29.3773 = 88.53% of the variance of the preference scores; hence, 
the error incurred by ignoring the residual variance might not be considered 
important in practice. In this case, individuals 1, 5, 6, 8, and 10 could be 
accepted as the limited panel of judges without further attention to differ- 
ences among their preferences. The loss of information resulting from the 


exclusion of the remaining individuals is indicated by the difference in vari- 
ance given by (39): 


41(26.0079 — 23.4348) = 105.4971. 
E this difference as x? with 20 degrees of freedom indicates that the loss 
P { 


i ', however, that the marginal increase in 
accuracy gained from any additional i 


the added expense or inconvenience 


Adjusting the scale of the y; for the selected individuals according to 
(40) yields 


e. 


шь = 120g, + .3893g,;, + 5881g,;, + 2.0758g,;, 


as the optimal linear estimator of the w 
and between occasions. 


It is of interest to compare the coefficients of this estimator with those 


+ .8766gr0;% 


ix With unit variance within objects 


^ 
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derived from the correlation ratios for these individuals by prior assumption 
of unidimensional preference scores. The resulting coefficients, brought to 
the same scale, are 


7922, .3743, .5708, 2.6558, .9015. 


The close agreement of corresponding coefficients reflects the large 
proportion of variance in the preference scores accounted for by the first 
canonical variance. 

The values for the pastries baked from the five fats are determined by 
applying the optimal linear estimator to the mean preference scores: 


Fat 
1 2 3 4 5 
Value: 3.933 3.596 2.922 —2.597 —7.854 


"These values must sum to zero. 

"A The use of the facilities of the Agricultural Experimental Station of the 
niversity of Puerto Rico for preparing the numerical example is gratefully 

acknowledged. 
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AN EMPIRICAL EVALUATION OF MULTIDIMENSIONAL 
SUCCESSIVE INTERVALS* 


SAMUEL J. Messicxt 


UNIVERSITY OF ILLINOIS 


The multidimensional method of successive intervals and the method 
of complete triads are applied to similarity judgments of Munsell colors 
varying in brightness, saturation, and hue. Both methods yield configurations 
that correlate highly with the Munsell color structure. This validation of these 
scaling methods in an area of known dimensionality indicates their applica- 
bility for exploration in areas of unknown dimensionality. 


Several recent multidimensional scaling procedures (2, 7, 14, 18), based 
upon a Euclidean mathematical model (19), reveal the number and the nature 
of relevant dimensions in unknown areas. Empirical evaluations of the methods 
in areas of known dimensionality seem appropriate before applying them in 
in areas of unknown dimensionality. The field of color perception was selected 
for such validation, since the dimensions of brightness, saturation, and hue 
are well defined. 

Richardson (14) applied the method of triadic combinations to judg- 
ments of similarity among some Munsell colors, which differed in saturation 
and brightness but were of a constant hue; the results are reported as being 
in essential agreement with the Munsell scheme. Torgerson (17) applied 
the complete method of triads to judgments of similarity among nine Munsell 
colors, which were presumably of the same red hue but differed in brightness 
and saturation. His two-dimensional configuration was very similar to the 
Munsell system. Torgerson also compared complete triads with unidimen- 
sional paired comparisons using nine gray stimuli that differed only in bright- 
ness. The methods yielded unidimensional scales that were linearly related. 
These results can be looked upon as а “validation” of this multidimensional 
scaling method. Indications are that saturation and brightness, at least, 


can be represented adequately by а Euclidean model (18). - e 
Since the method of complete triads requires 50 many judgments, thi 
task becomes prohibitive with more than ten stimuli. In order | Lee qum 
this difficulty, а multidimensional method of successive intervals (1, 8) wai 
the Office of Naval Research contract N Gonr- 
t by funds from the National Science 
as an Educational Testing 


= 
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recently developed; this method requires fewer judgments per ний: 
thus permits the use of а larger number of stimuli. This is ап impor a i 
consideration in investigations of unknown areas, for although the dimen 
sionality does not necessarily increase with the number of stimuli, it is 
certainly limited by that number. 

'This investigation is an evaluation of the multidimensional method of 
successive intervals in the area of color perception. With the new procedure 
it is feasible to include wide stimulus variations, so the study is also a step 
toward a systematic multidimensional mapping of psychological color space. 

f the stimuli are chosen from the Munsell color system (11), a comparison 
between the Munsell scale values and those obtained from the multidimen- 
sional procedure will permit an evaluation of the method. Since Torgerson 
has already applied a multidimensional scaling procedure (complete triads) 
to color stimuli with acceptable results (17), a comparison of multidimensional 
successive intervals with complete triads can also be looked upon as a valida- 
tion procedure. One aspect of this comparison will be an investigation of 
the difference in difficulty of the judgments required. In complete triads, 
which presumably involves a much simpler judgment, the subject must 
decide which two of three stimuli are most similar, Successive intervals 
requires the subject to decide whether the members of pair A are more or 
less alike than the members of pair B and then to order the pairs accordingly 
on a continuum. The practical importance of this difference may be evaluated 
not only in terms of the similarity of the final structures but also in terms of 
the time required to make the judgments, the ease with which they are made, 
and their acceptability to the subject as reasonable tasks, 


The Methods 


The multidimensional scaling methods are techniques for estimating 
the interpoint distances am 


ong a set of stimuli. [For detailed discussion of 
these methods, see (9).] By requiring judgments about pairs of stimuli 
methods yield seale values which can be taken 
veen the members of the pairs (20). The general 


Which can be analyzed to 
' Projections of the stimuli on 
а set of axes placed in the space (10). 


Since each of n(n — 1)(n — 2)/6 possibl 
n(n — 1)(n — 2)/2 judgments аге require 
dimensional method of successive interv: 


е triads is presented three times, 
d from each subject. In the multi- 
als (1, 8), on the other hand, the 


ee > eee 8 ee 


н niui 


| 
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poe is asked only to arrange the n(n — 1)/2 possible pairs of n stimuli 
: ona 1) categories on a distance continuum according to the degree of 
es ud ra the K of each pair. This procedure is a direct extension 
nsional successive intervals (4 rith pai imuli i 
шд vals (4, 6), with pairs of stimuli substituted 


Experimental Procedures 
The Stimuli and Their Mode of Presentation 


d кел nine stimuli used by Torgerson were sealed by the method 
E Eny According to the Munsell system (11), these stimuli are 
M Е ри! differ in brightness and saturation. Eight additional 
cde b hs e same characteristics were scaled by multidimensional 
icd r е s. Although these colors are of the same hue according to 
кн О, on the report of the Optical Society of America subcommittee 
Fe peer abet e Munsell colors (13) indicates slight hue variations. Since 
e ne P 8! ions represent an attempt to obtain a closer approximation 
ü ppearing intervals than the spacing in the Munsell scheme (12), 
hey would seem to be superior as criteria. 

PAS. he OSA revised designations of „the 16 stimuli scaled by multidimen- 

al successive intervals are plotted in Figure 1. The circled values are the 


HUE R 


4 
CHROMA VALUE CHROMA 


FIGURE 1 
Stimulus Configuration According to OSA Revisions 


the method of complete triads. The Munsell 
е dimensions of color perception as hue, value 
(brightness), and chroma (saturation). Theoretically steps along the value 
. and chroma scales in the Munsell system represent equal sense distances 
along the respective psychological dimensions, two chroma steps representing 


а sense distance approximately equal to one value step. 
For the method of complete triads, sheets of colored paper obtained 


eight stimuli also scaled by 
notation designates the thre 
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from the Munsell Color Company were cut into equilateral triangles 1.5 
inches on a side. Each of the 56 combinations of three different colored 
triangles was then mounted on a white cardboard triangle 6.25 inches on а 
side. For the multidimensional method of successive intervals, the colored 
sheets were cut into one-inch squares, and each pair of different colored 
squares (120 pairs for 16 stimuli) was mounted on a white 3" X 5" card. 
Each stimulus appeared as often on the left as on the right. The cards for 
both procedures were randomly arranged for presentation to the subjects. 
The stimuli were viewed against the white background of the mounts, which 
in turn were presented against a gray background. The lighting source was 
a ceiling fixture containing two GE 40-watt fluorescent daylight bulbs. 


The Subjects 


Forty-two subjects, 38 males and four females, took part. Color-blind 
persons were excluded by tests. The subjects were randomly divided into 
two equal groups, members of one group judging complete triads first and 
members of the other group judging successive intervals first. 


Instructions 


For the complete triads task, the subjects were to decide whether the 
top color on a triangular card looked more like 
The response of “right” or “left” 


subjects were also told that the с 


the left one or the right one. 
was recorded by the experimenter. The 
| olors on some triangles were very much 
alike, but that no triangle had colors exaetly the same. "They were encouraged 
to сч with a first impression and not to spend too much time on any 
one triad. 


als task, the subjects were 
les. First they were to 
g to whether the colors 
en the subjects were to 
at any time a subject wished to 
е was permitted to do so. Then 
way, giving eight piles. The left-hand iiis four piles into two in the same 


i i : D, the subjects looked through 
each pile to see if the cards in that pile seemed to “go together” and to make 
any necessary changes. 


The Number of Judgments Required 


The number of judgments for multidimensional successive intervals 
is minimal if each subject looks at each stimulus card only once and then 


—— 


f. — + tC 
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decides in which pile it belongs. The.minimum number of judgments, then, 
is equal to the number of stimulus cards [n(n — 1)/2]. Use of this procedure 
would have required 120 judgments from each subject in order to scale 
16 stimuli. It was desired, however, to have the subject re-sort the cards 
after he had become familiar with them; using an expanded multidimensional 
successive intervals procedure, such check sorts would be obtained without 
the number of judgments approaching appreciably that required by complete 
triads. Since each of the n(n — 1)/2 cards was looked at four times, 2n(n — 1) 
or 480 judgments were made by each subject in order to scale 16 stimuli. 
If these 16 stimuli had been sealed by complete triads, however, n(n — 1) 
(n — 2)/2 or 1680 judgments would have been required. 

In the complete triads section of the present experiment, each of the 56 
triads was presented three times, making a total of 168 judgments required 
from each subject to scale eight stimuli. 


Analysis and Results 


The Method of Complete Triads 


The raw data for the complete triads section of the study consisted of a 
42 (subjects) X 168 (triads) table of responses. These data were analyzed 
by the complete triads procedure (18) to obtain a matrix of relative inter- 
stimulus distances. The general multidimensional scaling procedure (10) 
was then applied to obtain a matrix of projections, which is analogous to a 
factor matrix. Its rank, т, is equal to the dimensionality of the Euclidean 
space defined by the experimentally obtained distances, and its elements 
represent the projections of the stimuli on a set of r orthogonal axes placed 
at the centroid of the stimuli. 

The matrix of projections was of rank 2. A third dimension might have 
been expected to represent the differences in hue. Its absence may be accounted 
for by the fact that the hue variations were slight compared with the range 
of variations in brightness and saturation. It is also possible that eight 
variables were too few to distinguish a third factor from error. A detailed 
description of the analysis and a tabulation of the data appear in (8). The 

as rotated orthogonally to approximate the 


two-dimensional structure W ! r 
Munsell dimensions. The complete triad structure and the revised Munsell 
comparison of each triads factor 


structure agree closely, as can be seen in a со 
with the corresponding Munsell dimension (Figure 2). The present structure 
also had about the same degree of agreement with Torgerson’s results (17). 


The Multidimensional Method of Successive Intervals 
ive i ls method were summarized 
The raw data for the successive interva. l 
in a 120 (stimulus-pairs) X 8 (categories) table of the number of times the 
ith pair of stimuli was placed in the gth category. Some of these stimulus-pairs 
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SATURATION 
BRIGHTNESS 


FIGURE 2 
Separate Comparisons of Triad Dimensions 
vs. Revised Munsell Dimensions 


had been unanimously or nearly unanimously placed in one of the extreme 
categories, making it impossible to obtain a seale value for every pair by 
ordinary successive intervals procedures. If four stimuli (stimuli 6, 12, 15, 
and 16) were excluded from the set, however, the frequency distributions of 
the remaining pairs had satisfactory ranges. Т hus, the set of distances among 
the remaining 12 stimuli was scaled by the method of successive intervals. 

By the selection of two other sets of stimuli with satisfactory distribu- 
tions (a set composed of the ten stimuli 1, 2, 4, 5, 6, 8, 9, 10, 12, and 15 and 
а set composed of the six stimuli 4, 8, 10, 12, 15, and 16), it was possible to 
obtain distance estimates involving all 16 stimuli. АП possible distances 
among the 16 stimuli could not be obtained, of course, but it was hoped 
that there would be sufficient overlap among the three sets to allow the four 
excluded points to be fitted into the Space defined by the set of 12 stimuli. 
Since some of the distances were involved in all three sets, the similarity of 
overlapping estimates could be evaluated. If the common distances were 


similarly estimated in all three groups, it would seem reasonable to attempt 
to fit the excluded four points into the space defined by the set of 12 stimuli. 
When the three sets of stimuli we 


re analyzed separately, the common 
distances were found to be estimated similar 


ly in all three sets, Accordingly; 
the four excluded stimuli were fitted into t 


ed в he space of the 12 stimuli by ® 
least squares criterion (8). The results for the set of 12 stimuli only will be 


considered in this paper, but it is suffice to say that this procedure located the 
four excluded points in positions reasonably appropriate to the Munsell 
scheme. 

The raw data for the set of 12 stimuli, then, consisted of a 66 (stimulus- 
pairs) X 8 (categories) table of the number of times the ith stimulus-pair 
was placed in the gth category. Successive intervals procedures (3) yielded 
the inter-stimulus distances, and the general multidimensional scaling pro- 
cedure (10) was applied to obtain a matrix of projections, which consisted 
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of three factors. The three dimensions were rotated orthogonally to approxi- 
mate the Munsell configuration. The Munsell and the successive intervals 
configurations are correlated to an extremely high degree. The extent of the 
agreement is seen in a comparison of each successive intervals factor with the 


FIGURE 3 
Separate Comparisons of Successive Intervals Dimensions 
vs. Revised Munsell Dimensions 


corresponding Munsell dimension (Figure 3). Factor A corresponds to satura- 
tion, Factor B to brightness, and Factor C to hue. 

_ These comparisons reveal only three widely deviant values, the hue 
positions obtained by successive intervals for stimuli 1, 4, and 10, which were 
the most unsaturated colors used. When such stimuli are compared in close 
proximity to stimuli with obvious hue, contrast effects might exaggerate 
relative inter-stimulus distances. It is also possible, however, that these 
deviant values indicate a slight deviation from orthogonality in the place- 
ment of the reference axes. With oblique rotation procedures these three 
points may be re-aligned to produce a linear plot in Figure 3 for hue vs. 
Factor С. Such an oblique orientation for Factor C might be considered an 
indication that the orthogonal axis placement in the Munsell system is 
somewhat in error. This is only a very tentative suggestion, but it illustrates 
that multidimensional scaling procedures, being based upon cross-dimensional 
judgments, can lead to such statements about relationships between dimen- 
sions. On the other hand, these three deviant points in Figure 3 might also 
be taken to indicate a slight inadequacy on the part of the Euclidean model 
to describe psychological color data. 


lete Triads and Multidimensional Successive 
Intervals 


A comparison between the final rotated structures obtained by complete 
triads and multidimensional successive intervals indicated excellent agree- 
ment. The fact that three dimensions were obtained from the successive 


A Comparison Between Compl 
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intervals procedure and only two from complete triads is probably more а 
function of the stimuli than of any difference between the two techniques. 
Only half the number of stimuli in the successive intervals experiment were 
scaled by complete triads, and the variations in hue involved were slight 
compared with the saturation and brightness variations. 

The average length of time for the 168 judgments by complete triads 
was 35 minutes, whereas the average time for the 480 judgments in the 
suecessive intervals task was approximately 40 minutes. It is evident that 
the successive intervals judgment, although presumably more difficult than 
the triads judgment, was made with considerable ease. Every subject accepted 
the task in a casual manner, without complaining about its difficulty. 


Discussion 


Since multidimensional scaling procedures yielded structures which cor- 
related highly with the revised Munsell system, it would now seem reasonable 
to apply these procedures for purposes of exploration and discovery in areas of 
unknown dimensionality as well as for confirmation and modification in other 
areas of known dimensionality. The results of this experiment also indicate the 
desirability of a systematic multidimensional scaling of psychological color 
space and, perhaps, a modification of existing color scales. The Munsell color 
system and the OSA revised scales are based upon psychophysical investiga- 
tions of each dimension separately (12), a procedure which does not permit the 
comparison of color samples across dimensions. Since the multidimensional 
scaling approach does allow multidimensional comparisons, a systematic 
application of these techniques in the color domain should es the con- 


struction of a color solid which is based upon the relationships between as 
well as within dimensions. 


REFERENCES 
1. Abelson, R. P. A technique and a model for ltidi М i ý 
Psychologist, 1954, 9, 319. multidimensional attitude scaling. Amer. 


2. Attneave, F. Dimensions of similarit: E 

t л | y. Amer. J. Psychol., 1950, 63, 516-556. 

3. pou G. W, Messick, 8. J., and Tucker, L. R. A general least squares solution 
Е me intervals. Princeton: Educational Testing Service Research Bulletin, 

4. Edwards, A. L. The scaling of stimuli b i 

y the method i . 

ЖОЛОБО of successive intervals. J. Appl 

5. Guilford, J. P. The method of paired comparisons as a i l 
peo 4 o psychometrie method. Psychol. 

6. Gulliksen, H. A least squares solution for successive 
standard deviations. Psychometrika, 1954, 19, 117-139. 

7. Klingberg, F. L. Studies in measurement of the relations b i tes. 
Psychometrika, 1941, 6, 335-352. Heu inopi pm 

8. Messick, S. J. The perception of attitude relationships: a multidimensional scaling 
approach to the structuring of social attitudes. Ph.D. thesis, 
Also Educational Testing Service Research Bulletin, 1954, { 


intervals assuming unequal 


Princeton Univer., 1954. 


r 


LS CMM „Ял RR 


20. 


SAMUEL J. MESSICK 375 


. Messick, S. J. Some recent theoretical developments in multidimensional scaling. 


Educ. Psychol. Measmt., 1956, 16, 82-100. 


. Messick, S. J. and Abelson, R. P. The additive constant problem in multidimensional 


scaling. Psychometrika, 1956, 21, 1-16. 


. Munsell Book of Color, Abridged Edition. Baltimore: Munsell Color Co., Inc., 1945. 


Newhall, S. M. The ratio method in the review of the Munsell colors. Amer. J. Psychol., 
1939, 52, 394. 


. Newhall, S. M., Nickerson, D., and Judd, D. B. Final report of the O.S.A. subcom- 


mittee on the spacing of the Munsell colors. J. opt. Soc. Amer., 1943, 33, 385-418. 


. Richardson, M. W. Multidimensional psychophysics. Psychol. Bull., 1938, 35, 659-660. 
. Saffir, M. A comparative study of scales constructed by three psychophysical methods. 


Psychometrika, 1937, 2, 179-198. 


. Thurstone, L. L. Rank-order as a psychophysieal method. J. ezp. Psychol., 1931, 14, 


187-201. 


. Torgerson, W. 8. A theoretical and empirical investigation of multidimensional scaling. 


Ph.D. thesis, Princeton Univer., 1951. 


. Torgerson, W. S. Multidimensional sealing: I. Theory and method. Psychometrika, 


1952, 17, 401-419. 


. Young, С. and Householder, A. 8. Discussion of a set of points in terms of their mutual 


distances. Psychometrika, 1938, 3, 19-22. 
Young, G. and Householder, A. S. A note on multidimensional psychophysical analysis. 
Psychometrika, 1941, 6, 331-333. 


Manuscript received 9/1/55 


Revised manuscript received 11/14/55 


PSYCHOMETRIKA—VOL. 21, No. 4 
DECEMBER, 1956 


A NEW SCALING TECHNIQUE FOR ABSOLUTE JUDGMENTS 


Burton S. ROSNER 


WEST HAVEN V. A. HOSPITAL AND YALE UNIVERSITY 


The results of an experiment using the method of absol j 
can be viewed as a dte of Conditional probabilities pA pena 
tepresent stimuli and the columns responses. The cosine of the angle between 
m row vectors is а measure of the similarity of the corresponding stimuli. 
4 cosine provides the basis for a method of scaling the stimuli. Unlike the 
a od of paired comparisons, this new technique does not require arbitrary 
xing of a unit of measurement. A numerical example is given. 


In the method of absolute judgments (2), the experimenter selects a 
sample of № stimuli S; (i = 1, ++- , N). He presents each stimulus singly 
to the subject; the order of presentation is randomized. The subject has 
available a sample of M responses R, (k = 1, --- , M). Each time a stimulus 
appears, the subject must make one and only one response. During the 
experiment, every stimulus is presented a number of times. Thus, the experi- 
mental data can be displayed as a matrix of conditional probabilities p,(k). 
This matrix has N rows, each row corresponding to a stimulus, and M columns, 
each column representing a response. The entry in any cell (ik) gives the 
conditional probability of №, given S; . Since the subject must make one 
and only one response on each trial, the sum of the probabilities in any row 


is 1.00. 
Each row of the matrix of conditional probabilities is a vector in M-space. 


This fact suggests using the cosine of the angle between two row vectors as 
a measure of the similarity between corresponding stimuli. The cosine of 
the angle ¢;; between row vectors and j, s;; , is given by 


x p.(5)p; (9) 
в = COS oi; = Ур" 02009 


Since each р: (К) is greater than or equal to zero, s;; will be positive and 
will lie in the interval 0.00 to 1.00, inclusive. A value of 0.00 signifies that 
none of the responses made to S; is ever made to S; and vice versa. A value 
of 1.00 signifies that the distribution of responses to S; is identical with the 
distribution for S; . Values between the upper and lower bounds indicate 
the extent to which the same responses are made to both stimuli. 

The measure of similarity between stimuli, S: , is in part formally 
identical with the Pearson product moment correlation—the product moment 
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correlation is also the cosine of the angle between two vectors. In the latter 
сазе, the dimensionality of the space containing those vectors equals the 
number of joint observations in the sample. The reliability of the product 
moment correlation depends upon the number of joint observations. But 
the reliability of s;; depends upon two factors: the number of observations 
available for determining each p,(k) and the total number of different re- 
sponses R, made to S; and S; . Responses made to neither stimulus do not 
help determine s;; . Thus, a sampling distribution for s;; would be very 
different from that for the Pearson r. Furthermore, s;; assumes positive or 
zero values only. Therefore, s;; differs from the Pearson r in several critical 
properties. 

Since s;; increases as the stimuli become more "similar" in the sense of 
evoking similar distributions of responses, there should be an inverse relation 
between the psychological scale separation of the stimuli and s;; . Drs. Bert 
F. Green and Robert P. Abelson have pointed out how this relation can be 
derived. The derivation leads to a scaling technique which does not require 
arbitrary fixing of a unit of measurement, as is necessary in the method of 
paired comparisons (4). This new technique naturally has many features in 
common with previous sealing methods for absolute judgments (1, 2, 3). 


Assume that the responses to S; are normally distributed, with a mean 
at the actual position of the stimulus. Thus, 


pk) = vx = ер E (EI d, , (2) 


where №, and Š; are the scale values of R, and 8,, respectively. To obtain 
bor pi(k)p;(k), use a continuous approximation: 


X» = [ 1 e |-3(6—5)] 1 


2 
"sss [i5] + (BSS) an. 


This expression is integrated by completing the square in the exponent. 
For convenience, let : 


A = (1/9) + (1/53), 


(4) 
В = (Si/e)) + (8/02), 


(5) 


and 


D = (8;/o%) + (8,/оў). (6) 


et 


une Ў eal 
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Substituting m (4) through (6) in (3) and completing the square, 
Epp) = zi ] c» (M4  — D/ Vay 
— (D/ V AY) dfi, 


e iO; 


— 0V (7) 


210,0; 


LI 


А | exp (3408 — Р/АУЎ} dfi 
E VEL e» 1-08 - ФГУЗ? n VY 


By resubstituting from equations " through (0) and simplifying, (7) becomes 


Xp = STS ыле Г (8, = 5. " 


If S; = S; , then (8) reduces to 
1. 
A) --—-——: (9) 
Э 2 ( ) ут с; 
Thus, evaluating the discriminal dispersions directly: 


Mo T (10) 
` 24/7 x p(k)" 


Finally, taking logarithms to the base e on both sides of (8) and simplifying 

the resulting equation using (1) and (9), 

(8, 8) = (ei + 0) 0108 2 + log c; + log v; (11) 
— log (ci + 07) — 2 log cos Ф]. 


When all diseriminal dispersions are set equal to unity, (11) becomes 
8,— 8, = 2У —log eos фа - (12) 


Reduction of ex toa iiti formula yields 


А L(t 1 ) 
(S: — 50 = ae GE Ft Tew 


[toe 2 - e (уст, or тив) ч 


— 2 log > ind 
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Mean scale separations between stimuli can be obtained by the usual tech- 
niques and can be summed to yield scale values for the stimuli. 

А "То illustrate the use of this scaling method, a modification of an ex- 
periment done by W. J. McGill was performed. A single subject sat before 
а 3-foot square black panel. An opal glass plate, 10 inches long by 4 inches 
wide, was mounted in the middle of the panel. Every 10 seconds, an 8-inch 
long horizontal line appeared on the plate for 1 second. А vertieal marker 
on each line extended 1/2 inch above and 1/2 inch below the line. The position 
of the marker varied from trial to trial. The marker could appear at the 
midpoint of the horizontal line or 1/2, 1, 1-1 /2, or 2 inches to either side of 
the midpoint. The subject registered judgments of the marker's position by 
depressing and then returning to normal one of fifteen toggle switches. The 
switches were mounted in a 14 by 2 by 2-inch box resting in front of the panel. 
Each stimulus was presented 110 times, but the first 30 trials were not used 
in scaling the stimuli. 

Table 1 shows the matrix of conditional probabilities obtained for this 
experiment. The inter-stimulus distances computed from (13) appear in 
Table 2. Table 3 shows estimated distances between stimuli S; and 5. 
These distances were obtained by subtracting from each non-zero entry of 
Table 2 the corresponding entry in the next column to the right. The sums 
and means of the columns of Table 3 appear at the bottoms of the columns. 
Finally, by setting S, at 0.00 and cumulating distances between adjacent 
stimuli, the scale values shown in the lower half of Table 3 are obtained. As 
should be expected, these values show that the subject discriminated poorly 
between adjacent stimuli on the extreme left or extreme right while discrim- 
inating the middle stimuli quite well. 
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DETERMINATION OF THE NUMBER OF INDEPENDENT 
PARAMETERS OF A SCORE MATRIX FROM THE EXAMI- 
NATION OF RANK ORDERS* 


ЈОЅЕРН F. BENNETT 


MASSACHUSETTS INSTITUTE OF TECHNOLOGY 


Two ordinal consequences are drawn from the linear multiple-factor 
analysis model. First, the number R(s, d) of distinct ways in which s subjects 
can be ranked by linear functions of d factors is limited by the recursive 
expression R(s, d) = R(s — 1, d) + (s — 1) R(s — 1, d — 1). Second, every 
set S of d + 2 subjects can be separated into two subsets S* and 8 — S* 
such that no linear function of d variables сап rank all S* over all S — S*, 
and vice Versa. When these results are applied to the hypothetical data of 
Thurstone's "box problem," three independent parameters are found. 
Helanons to Thurstone’s suggestion for a non-correlational factor analysis are 

Li . 


In the introduction to Multiple-Factor Analysis (3, p. xiii), Thurstone 
suggested that: 


“it would probably Ье... profitable to develop non-metrie methods 
of faetor analysis. An idea for such a development would be to determine 
the number of independent parameters of a score matrix by analyzing 
successive differences in rank order on the assumption that they are monotonic 
functions of a limited number of parameters. A score would then be regarded 
as merely an index of rank order, and that is essentially what we are now 
doing. The raw scores are transmuted into a normal distribution of unit 
standard deviation, and these transmuted scores are used for the correlations. 
Instead of dealing with the transmuted scores in this manner, one might deal 
. . The actual numerical values are of secondary 
importance in teasing out . . - the underlying order of a new domain. ... In 
putting the results to practical use, the problem would return to a metric 
form, with standardization and norms which call for statistical methods 


of the conventional kind.” 


with the rank orders directly . 


the suggestion interesting, but in the present 
between ordinal and metric measure- 
r such a non-metrie approach would 
t correlational methods, or indeed 


Workers in the area find 
state of knowledge about the relation 
ment it is impossible to tell whethe 
have practical advantages over presen 
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whether it is even possible in theory. This paper, to forestall any initial 
misunderstandings, will not attempt to answer these questions. Its purpose 
js to point out two particular ordinal consequences of the factor analysis 
model which became apparent during the development of a generalized 
form (1) of the “Unfolding Method" of Coombs (2), to show how these 
consequences can be applied in factoring hypothetical data, and to provoke 
interest in the problems which would have to be solved in applying such 
methods to empirical data. Ё 


General. Considerations 


Suppose that the factor and population matrices are given. Let the 
factor scores of each subject be taken as the coordinates of a subject-vector 
in a population space, the axes of which represent a set of orthogonal factors. 
(& subject-vector will refer specifically to its terminus; 
would do as well. For example, the subspace generated by two such vectors 
will be regarded not as the plane spanned by them and including the origin 
but as the line connecting their termini and, in general, not containing the 
origin.) It is possible to represent a test in such a sp 


ace as a line through the 
origin of the space with appropriate direction cosines relative to the axes, 


such that the orthogonal projections of the subject-vectors on a test-line are 


the subjects’ scores on that test, up to a linear transformation; the order of 


those projections on the test-line is the order in which that test ranks the 


the term subject-point 


subject-vectors, A and В. Wh 
A over B from those which rank 


, d ""-- 

they shall pass through the origin restri ошап ks 
A test-line coincident with H(A, B) wi 
ments which follow the convention i 
identical scores to two subjecty is no 
faction of two rankings, A > B an 
important in referring to the number of distinct rankin; 
are two rankings, A > B and B > A, but thr 
measurement, А > B, B > A, and A = B, 


gs. In Figure 1 there 
ee possible consequences of 
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FIGURE 1 


FIGURE 2 


Next consider the three subject-vectors in Figure 2. Every pair of 
subject-vectors will generate a line through the origin which is the unique 
line on which their orthogonal projections coincide. The three lines so formed 

will together partition the space into six regions. For example, any test-line 
into the shaded region shown will rank A over B and B over C. These lines 
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i i 1 f the space, because the regions 
г termine the ordinal properties о j c сал 
ved DE form correspond to all the possible rankings which a test could 
i the subjects. . 
ud pen example, consider the four subject-vectors of Figure 3. There 


FIGURE 3 


is again one line-of-coincident-projection for each 


pair of subject-vectors, 
or six such lines in all. Together they divide the space into twelve regions, 
each r 


egion having the property that any test into it will rank all the subjects 
in the same order. This means that when exactly two factors are present, 
there are only twelve different ways in which linear functions of those two 
factors can rank four subjects, and six of these rankings are exact inverses 
of the other six. (Note that regions which lie opposite to each other across 


the origin correspond to inverse rankings.) Now algebraically there are 
twenty-four different ways to rank four subjects; in this construction half 
of them are missing. This rai 


ses two questions with which to begin our 
investigation: why twelve rankings exactly, rather than ten or fourteen, 
and why these twelve? 


The Number of Linear Rankings 
The first question is essentially: how many different ways can s subjects 
be ranked by linear functions of d variables? 


Briefly, the answer is as follows: 
suppose this number is already known for some given number of dimensions 


and subjects, and it is asked what happens if one more subject is added. 
One new hyperplane will be formed between each subject already present 
and the new subject, and each of these hyperplanes will create as many new 


* 
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regions as it transects, since it cuts each of them into two. The problem is, 
therefore, to count the number of regions which each hyperplane transects 
and to multiply by the number of new hyperplanes. The former number can 
be determined by examining the surface of each hyperplane and noting the 
cells into which it is divided by its intersections with the other hyperplanes; 
each such cell will correspond to a region through which it has passed. But 
how сап these cells be counted? For the answer the author is indebted to 
Dr. Kenneth Leisenring (personal communication), who pointed out that 
the division of each hyperplane into cells by its intersections with other 
hyperplanes will be identical with the division into regions of the complete 
population space generated by one fewer subject and one fewer dimension. 
The reason for this identity is that each hyperplane is a legitimate subspace 
of the population space (that subspace from which all the variance attri- 
butable to the axis normal to the hyperplane has been extracted). АП lines 
on its surface created by its intersections with other hyperplanes are where 
they ought to be if all the other subject-vectors were projected normally 
onto its surface and the hyperplane were treated as a population space in 
its own right. In this projection the two subject-vectors determining the 
hyperplane will be projected onto а single point, because the hyperplane 
was originally constructed perpendicular to the line connecting them; hence 
the loss of one subject as well as one dimension. So it is seen that the number 
R(s, d) of regions or rankings created by s subjects in d-space is equal to 
the number present when the sth subject was added, that is, R(s — 1, d), 
plus the number of hyperplanes added (s — 1) times the number of regions 
which each transects, R(s — 1, d — 1). The result is the recursive expression 
shown at the top of Table 1. This account of its derivation is rather terse 
because the fruit of all this labor proves to have little except theoretical 
interest. R(s, d) quickly becomes large for moderate s and d, and so many 
tests are never likely to be given that the limitation on the number of different 
ways the subjects can be ranked becomes an important constraint on the 
data. It is useable directly only in the rare situation in which there are à 
large number of rankings of à small number of objects, as in psychophysics 


or in sealing experiments. 
Algebraic Properties of the Permissible Rankings 


The second issue is the constraint on the nature of the рег ү 
rankings imposed by dimension, which arises In the oen up- 
pose that three subject-vectors all lie on à straight line (which oes p pes 
sarily pass through the origin) and that on this line E^ are in аа 
A, В, С. 8uch а configuration is illustrated in Figure 4. It is irem na 
regardless of the dimensionality of the space in which these ves ors gen D 
there are only three ways in which any linear functions of the axes can ran 
them: A, B, €; ©, B, A;and А — B = C. In other words, if B is between 
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TABLE I 


uber R(s,d) of Different Ways in Which S Subjects 
di er Be eed By Linear Functions of d Factors 


R(s,d) - (5-1,8) *(s-1)R(c-1,d-1) 


а 


1 1 1 а; 
i 2 2 2 2 
3 6.3 6 6 6 
k 12 ae bt ЕЯ 2 
5 20 12 120. 5! 120 
6 о їїг 180 720 . 6: 
1 i; 352 1,512 3,600 
8 56 66 3,976 15,184 
9 T2 1,09% 9,144 15,992 
10 90 1,72 18,990 Я 

u 110 2,62 36,410 318,188 
12 132 3,852 65,472 718,698 
13 156 5,436 111,696 1,504,362 
14 182 7,06 182, 364 2,956,420 
15 210 10,012 286,860 5,509,506 
16 20 13,162 437,040 9,812,406 
17 | m 17,002 647,632 16,805,046 
18 | 306 21,626 936, 27,814,790 
19 | 32 21,13% 1,325,934 4h, 674,778 
20 380 33,632 2,841,480 © ,867, 52h 
a | sæ 41,232 2,515,120 106,697,124 
22 | 462 50,052 3,319,992 159,493, 644 
23 506 60,216 581,136 233,853,468 
% | 552 71,854 5,866,104 336,919,596 
25 | 600 85,102 7,590,600 477,706,092 
26 | 650 100,102 9,718,150 

2 702 117,002 12,320,802 

28 | 196 135,956 15,079,856 

5 19,286,624 

io 


23,843,220 


i of the interval A-C, no ranking based on 
а linear function of the coordinate axes is possible in which the subject B 


; C). Consider next the four coplanar 


in the interior of a simplex for 
of the coordinate axes can g 
wholly separated from the se 


t of subject S. 
This relation, however + 


: f dimension d con- 
taining a particular set of d +1 ] j 
randomly scattered so that no two of 


them are coincident, no three form а 
line, no two pairs form parallel lines, 


ete. Consider next two such subspaces 


Bas 
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FIGURE 5 
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generated by two distinct subsets of subject-vectors. From familiar geometric 
considerations, the sum of the dimensions of two intersecting subspaces is 
equal to the sum of the dimensions of their union and their intersection. 
For example, two one-dimensional lines crossing in a plane have the whole 
two-dimensional plane for their union and intersect in a zero-dimensional 
point. In that case, one plus one equals two plus zero. Now suppose that the 
total number of subject-vectors in the two subsets is equal to d + 2, where 
d is the dimensionality of the population space. One of the subsets, containing 
n subjects, will generate an (n — 1)-dimensional subspace. The other, con- 
taining d + 2 — n subjects, will generate a (d + 1 — n)-dimensional sub- 
space. The sum of these dimensions is d. On the other hand, the dimension 
of their union is clearly d also, since any d + 1 of them are sufficient to 
generate the whole population space. It follows that the dimension of the 
intersection of the subspaces is zero; that is, they must meet in à point. 
Such a configuration is illustrated in Figure 6. The two subject-vectors 


Ficure 6 


A and B generate a line AB which intersects a second line CD. In this par- 

ticular case, the point of intersection is between А j i AB 
"uon a 

and between С and D on the line CD. Iti шү л on ihe DoE i 

of intersection as a sort of phantom subje 

of subject-vectors and must obey appr: 

For example, in Figure 6, no linear fun 


} 
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the ranking XAB or ABX. Only AXB and ВХА are possible. The same is 
true of the line CD. Furthermore, the phantom subject must continue to 
obey these constraints relative to each subset when the two subsets are 
combined in any ranking. For example, in Figure 6, the ranking ABCD is 
impossible, because it would call for the point of intersection X to be between 
A and B and at the same time to be between C and D, which is impossible. 
On the other hand, the order ACBD is possible, because it can be written 
ACXBD, placing X in the middle of both subsets, АВ and CD. The funda- 
mental constraint which is imposed on rankings is that no ranking is possible 
in which this phantom subject is called upon to be in two places at once. 

There are clearly several different ways in which a given set of d + 2 
subjects can be separated into two subsets. Of these, the most important 
is that which arises when the point of intersection is in the interior of the 
simplex in both subspaces. That is, if one of the subspaces is a line, the 
point of intersection is between the two points which determine the line; 
if one of the subspaces is a plane, the point of intersection is inside the tri- 
angle formed by the three points determining the plane; and so forth. The 
uniqueness of the separation of the subjects into two sets which accomplishes 
this leads to the following proposition: If the population space is of dimen- 
sion d, then given any set S of d + 2 subjects, there exists a unique separation 
of the subjects into two subsets, S* and S — S*, such that no test can rank 
all the subjects in S* over all the subjects in S — S*. That is, no ranking 
can separate the two sets entirely, because then the phantom subject, the 
point of intersection, could not be inside both of them at once. 

The simple cases which have been discussed, for example, that of the 
vector which lies in the interior of the triangle formed by three other vectors 
in a plane, are merely special cases of this general proposition in which one 
of the subsets contains but a single vector. 

This result is applicable, with obvious limitations, to the problem of 
determining the number of independent parameters of the score matrix. 
The score matrix for subsets of n subjects is examined. If 4 independent 
parameters are present, at least some subsets of subjects of size d + 1 а 
separated іп rank by the tests in all possible ways; and, if there are enoug 


tests, all such subsets will be separated in all possible ways. But in every 
subset of d + 2 subjects there will be one separation which never Mic in 
the data, no matter how the elements of each subset are panir ыз. m 
the subset. This exclusion indicates that all the rankings can be accounte 


for by d independent parameters. 
Examples from the Box Problem 


Finally, examples of the exclusion property will be zs from ау 
matrix of Thurstone’s box problem (3, p. 140), to "m Е a = = ө 
number of independent parameters can be pests k ah c d 
scores on tests 1, 2, 4, 7, 13, 14, 18, and 19, which are known to co y 
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the factors X and Y. Of course when the scores are ranked the distinction 
among tests 1, 13, and 18 disappears at once because they are all monotonic, 
though non-linear, functions of X. The same is true of tests 2, 14, and 19. 
It is obvious that there must be more than one parameter present, because in 
many instances subsets of three subjects are separated in all possible ways. 
For example, boxes 1, 3, and 6 are variously ranked in the order 3, 6, 1 by 
test 4, and in the order 6, 3, 1 by test 7; no possible way of separating these 
three boxes into two subsets is absent. On the other hand, for every set of 
four boxes, there is some separation which no test imposes. For example, 
boxes 3 and 6 are never ranked apart from boxes 1 and 8. Box 8 is never 
ranked apart from boxes 1, 11, and 14. Thus two factors are sufficient to 
account for the score matrix. Proceeding to the analysis of all twenty tests 
together, it will be discovered that some sets of four subjects are separated in 
all possible ways. For example, boxes 1, 10, 11, and 14 are variously ranked 
in the order 10, 11, 14, 1 by test 17; 14,10. T1, 1 by test 8; and 11, 14, 10, 1 
by test 7. Every possible way of separating these four boxes into two subsets 
is represented. But in every set of five subjects а separation is discovered 
which does not occur in the data. For example, boxes 3, 5, and 7 are never 
separated from boxes 2 and 9. Therefore three fac 


tors are sufficient to account 
for all the rankings of the subjects generated by these tests. 


Conclusion 


broken down into their c 
that nearly all these pai 


] › With a requirement 
12 тей сотрагіѕопѕ shall 
deviation. 


Second, the determination of the number of di i 
imens ary to 
account for the whole set of tests d. paene d 
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